Sunday, August 23, 2009

The Principle of Peer Review

The recent posts by Joe discussing some of the major ideas of science got me thinking, so I decided to wade in on the conversation and add a few things that Joe may not mention. I apologize if this is a little long, but I was trained as a philosopher so I may tend to be wordy.

For many people their only connection or exposure to science comes from a high school science class (or college), or from science centered magazines such as Scientific American or Popular Science. For some people their only exposure to science and scientists in general comes from Hollywood or TV (shudder). For all those who don't have personal experience with scientific research there are a few critical things that never seem to get mentioned in any of these outlets. One of those critical ideas is the principle of peer review.

Peer review is something so fundamental to science that often scientists themselves forget to mention it when explaining what they do to non-scientists, much in the same way a congressman would not think about explaining the fundamentals democracy if someone asked him to explain what he does in his job. This comparison to a politician is apt and I will use it later on, but first let me dispel some misconceptions. The other day I was watching a TV show and as part of the story a "scientist" came and informed the people in a small town that they would have to move because their presence was threatening the local wildlife. As proof of her claim the "scientist" presented her "data" in the form of her doctoral dissertation, which was just a stack of paper in a clear plastic binder. Through out the episode several characters keep referring to the "data" and how it "speaks for itself". Each time someone said something like that my skin began to crawl. Also through out the episode the "scientist" continues to assert that she is right because she has "the data" and she is a "scientist" (it turns out she was wrong, but anyway...). At about that point in the show I about collapsed on the floor in agony.

While scientists do tend to deal with data, the stereotypical scientist that we see in movies or on TV is so far from the truth that for those of us who actually are scientists it can be almost physically painful to watch. One of the things that made the TV show that I mentioned so bad was its complete disregard and ignorance of the concept of peer review, which is something so absolutely fundamental to science that without it science appears no more rational than horoscopes and astrology. To put the concept of peer review in proper perspective I will use an analogy to something we are all familiar with, government.

To put it simply peer review is like holding an election. When a scientist does work they study different phenomena and come up with ideas that explain what they observe. A critical part of the whole process is where the scientists take what they have learned and write it down and present it to others. This point of the process is like a politician campaigning for office, they need the approval of their peers in order to determine the future course of events, or public policy. At this point what the scientist has written is put to intense scrutiny and compared to previous published (peer reviewed) work and the personal experience and expertise of the reviewers. If (IF) what the scientist has written passes peer review then it goes on to be published. In a democracy the only person that gets to hold office is the one that actually gets elected. It is the same with science, the only articles that get publish and are then considered valid science are those that pass peer review. To insist that something is scientifically sound before it passes peer review would be like a politician insisting on moving into his office even before the election.

Once something has passed peer review and gets published, it does not automatically guarantee that it is scientifically sound. We don't declare a politician to be successful before they actually serve their term (despite the opinions of some people). The success of a paper or scientific principle is measured by how influential it is, and whether or not others can duplicate its results. This can be roughly measured by the number of times a paper is cited in other published papers. A large number of citations may indicate a successful paper and good science.

In the case of the TV show I mentioned at the beginning, one reason why its depiction of science was so bad was that the scientist made no mention of her paper being published, peer reviewed and backed up by additional work, all of which are necessary for something to be considered scientifically sound. The character's assertion that she was right because she was a "scientist" is kind of like a politician insisting that they are a democratically elected official because they were appointed by a king. It doesn't make sense and completely undermines the whole concept of democracy, or science as the case may be.

I wish to point out that the principle of peer review is not just a social convention that we use to conduct science. Peer review is useful in preventing bad science, and also in providing a mechanism to construct a useful scientific dialogue, but the reason for it is more fundamental than that. Peer review is an expression of one of the fundamental aspects of the philosophy of science. One of the fundamental driving forces behind science is the concept that everything we observe is independent of the observer. What this means is that if one person observes something then someone else should (or must) be able to observe the same thing. If not then it can not be considered scientific. Another way of putting it is that no one is a privileged observer that has access to knowledge and data that no one else does. This comes from a belief that the universe will always act in a consistent manner and that it does not arbitrarily change the fundamental laws of existence for any one person.

In this sense the principle of peer review is an expression of the fundamental belief that the universe is rational, consistent and does not give special status to any one person. Thus no one can be right because they are a "scientist" (or a preacher, or a teacher, or a politician, or a philosopher etc.), but they are "right" in as much as what they say is in line with reality. Because any one person can be mistaken in what they observe we provide a safe guard in the form of peer review. Thus the principle of peer review is both a statement about a scientists fundamental understanding of how the universe works and a check to make sure that the universe, and also we ourselves are honest and correct in what we have observed.


  1. What a great post!

    I agree, sometimes the "profoundness" of the peer review process goes unnoticed. The peer review process is an amazing thing when you contemplate all the ramifications you have done.

    You really made me think about why it would be such a fun thing for philosophers to muse over. If nothing else, it should be emphasized how crucial peer review has been to scientific progress.

  2. I enjoyed reading this article. I am very interested in reading about the philosophy of science. Are there any good books that you would recommend? Or could I just learn more by continuing to follow the blog?

  3. Ryan,

    You hit the nail right on the head! When I saw the title of your post I was a little skeptical - how interesting can it be to talk about peer review? But I realized about a paragraph in that you are exactly right. To me, peer review is a given. Science isn't cooked up in a secret lab by a rouge scientist, it's gradually decided upon by a consensus of informed observers. Science is science because it can be shown to be true by anyone with the time and interest to learn to do it properly.

    Now of course there are issues with the peer review process in that it often favors the old entrenched ideas over new ones, but by and large it gets the right answer eventually.

  4. Hmmmm...the philosophy of science. Ironically enough most of what passes as "philosophy of science" is not actually written by scientists, which means they generally get a lot of it wrong. Also ironically enough most of what is written about the philosophy of science is not peer reviewed. In some cases there may be a few editors that look it over, but philosophy of any type does not tend to be peer reviewed (there are some very good philosophers out there but ~75% of them give the rest a bad name).

    The best way to learn the philosophy of science is to read sceintific journals, magazines and books. Try to keep to those that are written by actual scientists, though there are a few notable exceptions. "The Structure of Scientific Revolutions" by Thomas Kuhn comes to mind. Also I have found that the magazine Scientific American tends to be very good and understandable by most. And of course there is this blog.

  5. Great post! While I agree that peer review is essential and that its pros far outweigh its cons, there is also the interesting side-effect that peer review occasionally causes important potentially beneficial theories to be rejected due to their apparent lunacy. Some theories were disregarded or passed over for some time because the current of scientific research did not favor them at the time they were presented, only to be rediscovered later. Wish I could think of a good example at the moment.

  6. Carl,

    One good example was Chandrasekar's mass limit for white dwarfs, which was openly riddiculed by Eddington in the 1920's. Eddington, in fact, was so personal in his attacks that he caused Chandra to leave Cambridge and move to the University of Chicago.

    While I do agree that peer review does have some issues (much like democracy), it's the worst system for conducting sceince, except for everything else.

  7. Carl, another example is continental drift, the idea the continents move do to plate tectonics:

    Even though Alfred Wegener proposed the idea in 1912, it was so severely attacked by fellow geologists that it didn't gain acceptance until the 1950s.

    Dr. Taylor at BYU tells the story saying here is one example where scientists were so biased that it science had to wait for an entire generation to die off before the true theory could be accepted. As Planck said to the effect "scientific progress sometimes only happens one funeral at a time."

    However, though it took a new generation or too, the true science was ultimately accepted. Though good science can be rejected by the entire community for a time, if there is something to it it always seems to finally come around. Even if the only hope is the nay-sayers have to finally die off.

  8. As clumsy as it is, it seems that this system would keep the bonified crackpots from stealing some temporary credibility.

  9. Stan, you're right it does. For every historical "failing" of the peer review system, there are 1000s if not more successes.

    Almost every day, either on arXiv or some conference etc... there is a crackpot promoting a crazy theory. Luckily, the peer review process quickly prevents such theories from spreading.

  10. Joe what you are mentioning is precisely what Kuhn covers in his book, The Structure of Scientific Revolutions. Not all scientific progress happens in this way but some (not all, or even most) of the major advancements came about in this way (Poisson's spot, Black Holes, Plate Techtonics etc.).

  11. Based on personal experience, peer-review can be a cut throat and nasty affair, but it is an essential part of science. I liked the analogy to politics and government.


To add a link to text:
<a href="URL">Text</a>