Pages

Wednesday, September 15, 2010

"Rational" Referees May Hurt The Peer Review Process.

ResearchBlogging.orgThose of us who work in academic fields hope that the peer review process in some sense works.  Thurner and Hanel recently studied the effects of one particular entity that may hurt the process: the rational referee.  Here, rational referes to someone who largely accepts or rejects a paper by factoring in how the acceptance or rejection of the paper will impact himself/herself. (To me this is not the best definition of rational which is why I have put it in quotes "" in the title.  However, I am reporting their work so I will stick to their definition.)

From the authors:
A fundamental problem of the peer review process is that it introduces conflicting interests or moral hazard problems in a variety of situations. By accepting high quality work and thus promoting it, the referee risks to draw the attention to these ideas and possibly away from her own. A post-doc looking for his next position is maybe not happy to accept a good paper of his peer who competes for the same position. A big-shot in a particular field might fear to risk his ’guru status’ by accepting challenging and maybe better ideas than his own, etc. In other words, referees who optimize their overall ’utility’ (status, papers, fame, position, ...) might find that accepting good scientific work of others is in direct conflict with their own utility maximization. In the following we call utility optimizing referees rational.
Though one might argue that it is obvious that self interests would hurts peer review, we would like to be able to put some numbers behind the idea so as to "quantify" how bad the problem might be.

To test the effects of a rational referee, the authors ran several simulations where papers are refereed by 3 types of referees described below.  The scientific quality of the papers follow a Gaussian distribution, ie each paper "is assigned an 'IQ' index... drawn from a normal distribution" with mean=100 and standard deviation = 10.  In addition to being a referee, each person in the simulation is also an author of a paper himself/herself but never referees their own paper.  Here are the types of referees considered:
  • The correct referee: This person is competent to judge the quality of the work, and only accepts the best scientific papers given.  (Using an algorithm described in the above paper.)
  • The stupid referee: This is someone who is not competent to properly judge the work and so the acceptance or rejection is random. (Who hasn't run across this? :) )
  • The rational referee: This is someone who compares the quality of the paper they are refereeing to the quality of their own work and accepts or rejects accordingly.
At this point it should be pointed out that if the average paper accepted has a score of 100, then the peer review process does no better than flipping a coin.  With that said, here are some plots:


The above plots show the results of average paper quality versus the fraction of rational referees.  The three separate colored curves are for different fractions of "stupid" referees.  For example, the blue curve has 10% of the referees being stupid.


The plot above here shows what happens after t publication rounds.  Fig. a is when all referees are correct. As you can see the average paper IQ is ~120.  Fig. b shows a histogram of the IQ of the papers accepted compared to the gaussian distribution they were drawn from.  Fig. c is the same as Fig. a except now 10% of the referees are rational and Fig. d is the same as Fig. b with the same caveat.  If only 10% of referees are rational, the paper quality diminishes significantly.

Conclusions:

The authors conclude thus:
The presence of relatively small fractions of ’rational’ and/or ’random’ referees (deviating from correct behavior) considerably reduces the average quality of published or sponsored science as a whole... systemic level. Our message is clear: if it can not be guar- anteed that the fraction of ’rational’ and ’random’ referees is confined to a very small number, the peer review system will not perform much better than by accepting papers by throwing (an unbiased!) coin.
 And as we all know, referees that factor in their own self interest or are incompetent to judge the works they are assigned exist.  As may be inferred from above, if these two groups aren't kept in check, the process may become no more reliable than flipping a coin.

Stefan Thurner, & Rudolf Hanel (2010). Peer-review in a world with rational scientists: Toward selection of the average E-Print arXiv: 1008.4324v1

12 comments:

  1. With modern science so specialized, I have a question. Of the people qualified to judge a paper in a given subfield (i.e. the not-Stupid referees), how many are not direct competitors of the authors of the papers?

    In other words is it possible that Not-stupid + Not-rational equals the empty set (or very close to it)?

    ReplyDelete
  2. Ben,

    You raise a very good point. I know for many of my papers the the very best people to referee them from a "competency" stand point are my competitors. :) (Or people in my "inner circle" who may be biased for other reasons.)

    ReplyDelete
  3. Sorry this is totally unrelated, but fyi Leonard Mlodinow is being interviewed tonight on Coast to Coast am at either 10 or 11 PDT. Joseph, that's kfi am 640 for you.

    ReplyDelete
  4. Stan,

    Thank you for the heads up and especially finding the station info for me. I'll have to tune in.

    ReplyDelete
  5. Oh, that's another good point. Collaborators would definitely fall into the "rational referee" category.

    ReplyDelete
  6. JS,

    Whatever happened to "peer" review? Now, if the field is so esoteric that there are hardly enough peers, well does it really matter whether world at large even know?

    Internet and blogs allow you "to publish" without peer reviews.

    For the record, this happens in all fields. Have seen a paper rejected because the idea had significant negative impact on the refree and the company. The rejection reason was lack of understanding of the experimental results by the author, even though the author went out of his way to provide at least three different mechanisms which may lead to the observation.

    ReplyDelete
  7. Anonymous,

    Thanks. And there is another issue: if a journal rejects your paper often you can find another journal to accept at which point your paper is "published".

    ReplyDelete
  8. It's a good topic. I'm a HUGE believer in the wisdom of crowds. In fact, I read a book about it not long ago called "The Wisdom of Crowds" by James Suroweiki. It was a really good book.

    I'm a huge fan of the peer review process. It's not perfect, and has its problems without doubt. And certainly it is subject to groupthink, biases, and a host of other ills. But I think it blows the alternatives out of the water any day of the week.

    To take Anonymous' concept to the next level, if it becomes too difficult to publish in peer reviewed venues, I suspect up and coming scientists WILL start publishing in free form venues. This will cause a paradigm shift and science will recover nicely. So personally, I don't worry much about the biases and nonsense that goes along with some academic journals. They will get their comeuppance naturally if they try to bias the results. The only regret will be how much good science they passed up by playing their game!

    The other point, to counter Anonymous' is that there need not always be enough "peers." If there is only one scientist working on one particular problem, then that scientist has the final say until more scientists start to solve the problems in that area. I don't view that as a problem, but as progress. Surely we can't have a plethora of experts in every new field immediately? These things evolve with time. A good example right now, is Andrew Newberg, a neuroscientist who studies why people believe in God, and how belief in God helps them. This field is attracting more scientists now, but he has been a pioneer in that area.

    OTOH, there might be a reason there are few peers in a field. There aren't many scientists studying apparitions and extraterrestrial phenomena because it has little credibility. Until someone produces good reason for us to investigate that area, a dearth of scientists will remain.

    ReplyDelete
  9. jmb275, very good comment and I want to say I actually also believe in the "wisdom of the crowds". I do think the ""mainstream" science communty will ultimatley converge on correct science.

    ReplyDelete
  10. The next obvious question is, can the readers of the journals then pick out the "smart" papers and cite those and not cite the "stupid" papers, thus putting all of the papers through another "peer review" process? This is of course problematic because the quality of the papers have already been reduced by the "rational" referees. Thus applying a second "filter" may not actually do much.

    Also as a side note, there is some indication that the dumbing down of papers has already happened in some fields. One place this has come up (unfortunately) is in climate papers, and especially the whole IPCC process. They apparently found systematic problems with their review process that they must address. While these systematic problems do not invalidate the bulk of the results, they do indicate that they are not as "smart" as thy could be.

    ReplyDelete
  11. Quantumleap42,

    I know a physicist here who one time chuckled: "Yeah, it turns out the paper has issues but at least it got a lot of citations." But I posted on a similar topic here.

    Just to be clear, scientists aren't up to no good. But sometimes the "human" in them comes out just like everyone else.

    ReplyDelete
  12. jmb275,
    Here is an article that may be of interest:
    http://www.wayofthemind.org/2006/09/27/why-do-people-believe-in-god/

    and,

    http://www.wayofthemind.org/

    in general, should be interesting.

    ReplyDelete

To add a link to text:
<a href="URL">Text</a>