The Principle of Peer Review

The recent posts by Joe discussing some of the major ideas of science got me thinking, so I decided to wade in on the conversation and add a few things that Joe may not mention. I apologize if this is a little long, but I was trained as a philosopher so I may tend to be wordy.

For many people their only connection or exposure to science comes from a high school science class (or college), or from science centered magazines such as Scientific American or Popular Science. For some people their only exposure to science and scientists in general comes from Hollywood or TV (shudder). For all those who don't have personal experience with scientific research there are a few critical things that never seem to get mentioned in any of these outlets. One of those critical ideas is the principle of peer review.

Peer review is something so fundamental to science that often scientists themselves forget to mention it when explaining what they do to non-scientists, much in the same way a congressman would not think about explaining the fundamentals democracy if someone asked him to explain what he does in his job. This comparison to a politician is apt and I will use it later on, but first let me dispel some misconceptions. The other day I was watching a TV show and as part of the story a "scientist" came and informed the people in a small town that they would have to move because their presence was threatening the local wildlife. As proof of her claim the "scientist" presented her "data" in the form of her doctoral dissertation, which was just a stack of paper in a clear plastic binder. Through out the episode several characters keep referring to the "data" and how it "speaks for itself". Each time someone said something like that my skin began to crawl. Also through out the episode the "scientist" continues to assert that she is right because she has "the data" and she is a "scientist" (it turns out she was wrong, but anyway...). At about that point in the show I about collapsed on the floor in agony.

While scientists do tend to deal with data, the stereotypical scientist that we see in movies or on TV is so far from the truth that for those of us who actually are scientists it can be almost physically painful to watch. One of the things that made the TV show that I mentioned so bad was its complete disregard and ignorance of the concept of peer review, which is something so absolutely fundamental to science that without it science appears no more rational than horoscopes and astrology. To put the concept of peer review in proper perspective I will use an analogy to something we are all familiar with, government.

To put it simply peer review is like holding an election. When a scientist does work they study different phenomena and come up with ideas that explain what they observe. A critical part of the whole process is where the scientists take what they have learned and write it down and present it to others. This point of the process is like a politician campaigning for office, they need the approval of their peers in order to determine the future course of events, or public policy. At this point what the scientist has written is put to intense scrutiny and compared to previous published (peer reviewed) work and the personal experience and expertise of the reviewers. If (IF) what the scientist has written passes peer review then it goes on to be published. In a democracy the only person that gets to hold office is the one that actually gets elected. It is the same with science, the only articles that get publish and are then considered valid science are those that pass peer review. To insist that something is scientifically sound before it passes peer review would be like a politician insisting on moving into his office even before the election.

Once something has passed peer review and gets published, it does not automatically guarantee that it is scientifically sound. We don't declare a politician to be successful before they actually serve their term (despite the opinions of some people). The success of a paper or scientific principle is measured by how influential it is, and whether or not others can duplicate its results. This can be roughly measured by the number of times a paper is cited in other published papers. A large number of citations may indicate a successful paper and good science.

In the case of the TV show I mentioned at the beginning, one reason why its depiction of science was so bad was that the scientist made no mention of her paper being published, peer reviewed and backed up by additional work, all of which are necessary for something to be considered scientifically sound. The character's assertion that she was right because she was a "scientist" is kind of like a politician insisting that they are a democratically elected official because they were appointed by a king. It doesn't make sense and completely undermines the whole concept of democracy, or science as the case may be.

I wish to point out that the principle of peer review is not just a social convention that we use to conduct science. Peer review is useful in preventing bad science, and also in providing a mechanism to construct a useful scientific dialogue, but the reason for it is more fundamental than that. Peer review is an expression of one of the fundamental aspects of the philosophy of science. One of the fundamental driving forces behind science is the concept that everything we observe is independent of the observer. What this means is that if one person observes something then someone else should (or must) be able to observe the same thing. If not then it can not be considered scientific. Another way of putting it is that no one is a privileged observer that has access to knowledge and data that no one else does. This comes from a belief that the universe will always act in a consistent manner and that it does not arbitrarily change the fundamental laws of existence for any one person.

In this sense the principle of peer review is an expression of the fundamental belief that the universe is rational, consistent and does not give special status to any one person. Thus no one can be right because they are a "scientist" (or a preacher, or a teacher, or a politician, or a philosopher etc.), but they are "right" in as much as what they say is in line with reality. Because any one person can be mistaken in what they observe we provide a safe guard in the form of peer review. Thus the principle of peer review is both a statement about a scientists fundamental understanding of how the universe works and a check to make sure that the universe, and also we ourselves are honest and correct in what we have observed.