Tuesday, May 29, 2012

Can peer review be improved?

Last week I was back in Oxford briefly and visited my DPhil supervisor Subir Sarkar, who, in the course of a long conversation about various things, told me about a discussion he had apparently had with Professor Anton Zeilinger after a recent colloquium there. Zeilinger had apparently mentioned the academic peer review process and had some ideas about how the process might be made to serve science better by being more transparent. I found these ideas (as relayed to me) rather thought-provoking, and I'd like to promote a bit of discussion about them on this blog.

To paraphrase the argument: an important problem with the peer review system as it stands is that in many (or even most) cases there is only one referee who decides whether the submitted paper meets the scientific standards of the journal in question, this referee is completely anonymous, and his or her report on the paper remains entirely confidential, seen only by the journal editor and the author(s) of the paper in question.

Guaranteed anonymity is well known to alter people's behaviour on the internet, in general making people more opinionated, far ruder, and less receptive to reasoned discussion about alternative points of view. Unfortunately, scientists being only human, this is almost certainly also true to some extent in the peer review process. So removing the referee's anonymity might well improve civility in some cases, and you can see why senior academics might be in favour of it — though I notice that my feeling that even anonymous reviewers are more polite when the corresponding author on a paper is a senior figure rather than a graduate student or young postdoc seems to be shared by some others, so senior academics should have the least to complain about.

Removing referee anonymity is not a completely crazy idea: I'm told it is the norm in Scandinavian countries for the identity of the panels reviewing grant proposals to be known to applicants — though here there is some aspect of collective responsibility, within the panel there may only be a single expert on a particular field, so in practice this doesn't offer a lot of cover. If all authors and reviewers were perfectly objective, with no tendency to personal animosity or desire for retribution, and all had the same permanent job security, it might work. But they aren't, and they don't, so it can't. In particular, much reviewing is actually done by younger scientists, postdocs at the start of their careers, who may well one day be applying for a job to the author whose paper they are reviewing. The potential conflicts of interest don't bear thinking about. I'm pretty sure referees must remain anonymous.

Similarly, it is impractical to demand several referees for each submitted paper. In physics a few, very high profile, journals do use multiple referees, but usually only for short papers in a Letter format (i.e., less than about 5 pages long). Having more than one referee for each full-length paper at each journal would presumably increase the time taken to accept or reject submitted papers to unacceptable levels.

What about the third option? Why can't journals put a link next to the online version of accepted papers, that allows those interested to read the correspondence between the authors and the (still completely anonymous) referee? Knowing this correspondence was to be made public might cause referees (and, equally importantly, authors replying to the referee's criticisms!) to be far more civil. It would certainly help put an end to the unfortunate but not-entirely-unknown incidents of referees holding up publication on specious grounds until certain citations are added to the manuscript. I think it would also raise the level of debate. Journals of course would only provide such a summary of correspondence for accepted papers, but authors could perhaps, if they felt their paper was unfairly rejected, also put the correspondence online, say on the arXiv.

But as I see it, the most important argument in favour of such a model is that it increases the information available to the readers of the papers. I personally often discover some interesting preprints on the arXiv from fields somewhat beyond my expertise. When looking at the published version of the paper on the journal website, I'd like to know what changes the referee requested, and how the authors responded to the request: this would give me an idea of what the contentious issues are in that field, and help me to decide how much weight to assign to claims made in the paper. This sort of information ultimately surely helps the advance of scientific knowledge.

For instance, in the case of the recent dark matter argument I blogged about here, I'd like to know whether the referee for the first paper just didn't realise there was a problem with the authors' assumptions, knew about the issue but didn't think it was a problem, or knew that it was contentious but decided on balance the paper was interesting enough to be published anyway? If the referee simply missed it, does that tell us something about the standards of refereeing at the journal in question (a well-respected one in this case), or did the authors of the second paper have some unique insight?

At the moment I can see lots of positives but no drawbacks to this idea of providing more information about the process of peer review. Other people might have other ideas, and I'd like to hear from you in the comments — I'm happy to be convinced either way!

8 comments:

  1. I think in this post I made it sound a little too much as though the objective was to make the correspondence between referees and authors more polite. Actually I think the main advantage would be to obtain a better standard of refereeing, with more careful analysis, and better dissemination of information to readers of the papers. Hopefully that point wasn't lost in my writing!

    ReplyDelete
  2. All three of my papers to date have had multiple referees. I figured it was the norm - so what am I doing to deserve this!?

    ReplyDelete
    Replies
    1. Hi Phil. That surprises me: in my experience Phys. Rev. D, JCAP and MNRAS by default only have one referee. If there are any disagreements with the authors that can't be resolved, another referee may be called in to provide a second opinion. PRL normally has two referees, with a third referee only being called in if required to be a tie-breaker. Other 'Letters' journals are probably the same.

      Did you have multiple referees with PRD right from the start?

      Delete
    2. Yep, right from the start. With CQG too.

      Delete
    3. Ah well. Live and learn, eh? Thanks for the feedback.

      Delete
  3. All the papers I review (and have reviewed) have at least two reviewers, sometimes three. I terms of grant review panels these have also been available to me, but I do not know the exact members of the panel who represent my applications at the review panel meetings.
    I do agree that reviewers/referees seldom read a paper properly and have almost decided their opinion on gut feeling (and tend to request changes or make criticisms that have already been addressed within the paper).
    Sarah

    ReplyDelete
    Replies
    1. Hello Sarah. If only all physics journals had a similar default policy of more than one reviewer, some of these problems would be avoided! If this is standard in most other fields I don't know why it isn't the case for the specific journals I mentioned. An easy improvement to make, you would think.

      I totally agree about referees often making criticisms that have already been addressed in the paper - this is one of the most frustrating things to point out in replies to the journal editor! My line of reasoning was that if journals were as a matter of course to publish the correspondence where the referee says "I think the authors should mention X in their paper" and the authors reply with "Actually, we already said X in the abstract, on page 5 and again in the conclusions," this might put some public pressure on referees to actually read the paper carefully before commenting on it. Or if not that, then at least on the journal editors to stop picking such useless referees.

      Unfortunately while I can't see any scientific reason not to publish this correspondence, I can see that it is not to the advantage of any journal that wishes to maintain the illusion that its review process is objective and fair.

      Delete