Many of us have had the feeling that something is not right these days with the peer-review system in science. Whenever I chat with colleagues about the peer review system, two issues consistently crop up: an increasing number of review requests that we cannot possibly keep up with and/or reviews that seem to indicate a reviewer did not spend much time with the manuscript they were reviewing. So, when Ecology Letters published an article in 2008 (Hochberg et al), written by a group of its editors, titled “The tragedy of the reviewer commons”, I read with great interest. However, I was dismayed to see that apparently the entire fault for the current sad state of affairs lay with people like me: reviewers and authors. I was slightly peeved at the tone of the article that implied that things would improve if only reviewers/authors behaved better. Where was the responsibility of the journals/editors in this mess? I thought, “I really need to write a blog post on this”. I never got around to it. Since then, at conferences and in additional publications (e.g., McPeek et al 2008), I have heard the same refrains: Scientists need to review faster, better, smarter. I began to wonder if I was alone in this world in my feelings that reviewers/authors are only half of the equation. Then I read a blog article over at the Chronicle for Higher Education. This article was also about the problems with the peer-review system, but from the perspective of a reviewer/author. And I realized not only was I not alone, but that we needed more voices demanding real dialogue on this issue. So here we go: a reviewer/author’s take on how journals/editors can help reviewers/authors make journal/editors happier.
1) Better reviewer databases: I say no a lot to reviews because I say yes a lot to reviews, not because I lack a sense of scientific responsibility. The Chronicle blog (by a sociologist) points out that the number of members in the American Sociological Association is more than enough to support a reasonable number of reviews/person. However, a much smaller number of people seem to be shouldering the load. I suspect the same is true for ecology. So why is this? Undoubtedly the journals are right that there are curmudgeons who simply refuse to review. But I also suspect that editors are busy people like the rest of us and when we are busy we go with the names of people who come to mind quickly; these “go-to” people are “the most obvious people” to review a paper or give a talk. However, those go-to people are often the same for many people – resulting in the smaller number of people getting a higher load of review requests. As a reviewer I try to help with this situation by recommending people I think are not yet “in the system” (post-docs, young assistant professors, etc), but I might humbly suggest that journals invest in better reviewer databases to help editors come up with a better diversity of names.
2) More editorial control: My next two suggestions are not going to make me popular with either authors or editors. And I know (if they got implemented) I would occasionally get hoisted in my own petard, but I strongly believe that with the demands journals are making on reviewers theses days (thorough reviews, lots of reviews, quick reviews) journals have a responsibility to protect reviewers from superfluous reviews (i.e. unnecessary review requests).
a) Better pre-review vetting. Many authors will hate this because this means one person is probably deciding whether or not to send something out for review. A bad draw on an editor (who has a strong personal opinion on the validity/novelty of your work) can kill your submission. However, I am not alone in having received manuscripts for review that are so poorly written that they are in effect incomprehensible or so far from the journal’s standard that clearly no editor looked at the manuscript before sending it on to me. I’m not talking about borderline cases but manuscripts so bad I barely know how to review them. As a reviewer this just makes me mad and takes up valuable time that could have been dedicated to a manuscript that actually deserved consideration. As the Chronicle post, points out: manuscripts do not have a fundamental right to be reviewed.
b) Stop looking for reviewer consensus. I have noticed a trend at certain journals: manuscripts keep being sent back to the reviewer until the reviewer “signs off” on the manuscript. This is consistent with the idea in the Ecology Letters article that authors are needlessly lengthening the review process by ignoring reviewer comments. As much as we may all wish otherwise, not all reviewer comments reflect absolute truth. We all have our opinions on things that (if we’re being honest with ourselves) actually are in gray areas. Sometimes reviewers just flub things. And, journals are right, sometimes reviewers give shoddy reviews. As both a reviewer and an author I recognize this. As a reviewer, I assume the editor will read my review (and the paper) and decide for his or herself whether they agree with my opinion. As an author, I assume that the editor will read my response to a reviewer and decide whether my objections to a certain critique have merit. As a reviewer, the only time I want to re-review a paper is if I have labeled my concern as “fatal” and the editor is uncertain whether the authors have either dealt with that concern or have a valid argument for why it is not a concern. In a world where reviewers are scarce, manuscripts should only go back to reviewers when absolutely necessary. This requires editors to insert themselves more into the process than perhaps they have been accustomed.
Maybe journals and editors already feel like they do these things. I don’t know. I do know I feel like I already do the things they want me as a reviewer to do! However, given how widespread concern over the strain on the peer-review process is, it seems to me that perhaps it’s time for a real dialogue – and that involves both sides talking about their perspectives and making suggestions about how to improve things. Anyone out there have additional ideas for things that could be done?
Jabberwocky Ecology 
I couldn’t agree with this more. I actually received a review request this week from Ecology that included a link to the McPeek et al. paper along with the following sentence “As a journal dependent on the reciprocal nature of peer review, we kindly ask you to observe the golden rule of reviewing.” This actually had the opposite of the desired effect – it made me want to review the paper less. I review on the order of 20 papers per year, get them returned on time, and spend a lot of time working on each review. The implication that somehow if I didn’t review this paper for this journal, right now, that somehow I was being self-centered annoyed rather than inspired me.
Oh, and while I was looking up the above quote, I noticed that I’d received 3 review requests this week.
Pingback: No peer review crisis after all? | Jabberwocky Ecology