Jabberwocky Ecology

The peer review system has recently been under increasing pressure as the number of papers submitted has been skyrocketing. Jeremy Fox and Owen Petchey have recently proposed a new system for fixing this, so called, “tragedy of the reviewer commons.” The crux of the argument is that for every paper a researcher submits they must review three papers in exchange (thus balancing the review load imposed by each submitted paper). A centralized PubCred bank would keep track of reviews, submissions and the balance of credits for each researcher.

At first look this seems like kind of a cool idea and I’ve seen a recent surge of interest in it via email and an enthusiatic post over at i’m a chordata urochordata. However, there are, as I see it, two major challenges for this type of system. The first is that in order to make it function properly there have to be a bunch of detailed rules in place for special circumstances. The authors of the proposal address some of these and acknowledge that there will be others†. But who should make these rules? Certainly we won’t all agree on the best solutions (e.g., I think that forcing reviewers to rereview manuscripts without additional credit, as proposed, is dangerous and likely to lead to increasingly poor editorial practice*), so who decides. I,for one, would be loathe to hand this responsibility off to the publishers‡, so I guess we’ll need some sort of council of researchers, from across a breadth of disciplines and countries, preferably elected in some sort of democratic process and then they can meet and vote on the rules. That sounds difficult to setup and organize, but we are talking about a group that is going to control a major aspect of the scientific process, so we’d better do it right.

The other major challenge is setting up the actual technical aspects of the system. Fox & Petchey suggest that given currently available web technology that the basic setup should be no more than three person-months, and that sounds about right to me for the basic site. But it ignores some important complexities. The most serious of these is the lack of a universal author identification system. There are tens (if not hundreds) of thousands of individuals contributing to the writing and review of papers across disciplines and this will lead to numerous instances where authors/reviewers have similar/identical names. There are initiatives underway to address this problem (primarily motivated by search issues), but none of them is complete and we are probably years away from an agreed upon standard within disciplines (let alone among them). Until such a system becomes established it is difficult to understand how PubCreds could properly operate. I suppose the PubCred system could try to take on this responsibility itself, but I suspect that the political realities of numerous groups competing to provide this service will make that complicated.  In addition we would presumably need to consider a secure solution for validating payments from non-lead authors (in the proposal authors are allowed to split up the “cost” of review however they deem appropriate). If this is really going to be such a valuable currency as to solve the reviewer issue then it will be valuable enough to generate unseemly behavior. Maybe a simple email confirmation process would suffice, but we need something to prevent the lead author from unilaterally deciding on how to divide up the cost. Regardless, my point is simply that while the basic system is easy, if we are going to use this to literally govern whether or not a (potentially important) scientific paper can be submitted, then the system needs to be about as robust as a banking system, and accounting for complex contingencies and putting together appropriate security makes this quite a bit more than a 3 person-month job.

So I guess we’d better get to work, because to do this properly is going to take a lot of organization and some serious effort. Or, we could just “privatize the reviewer commons” in exactly the same way we “privatize” everything else. We could use money. This has already been proposed quite eloquently in an editorial by Lonnie Aarssen (and he even implemented this idea for a while at his new journal – IEE; see also the follow up editorial) that we’ve discussed here before. The current proposal discounts this possibility because:

…a fee-to-submit system would disadvantage authors who lack the means to pay, might require exorbitant payments in order to attract referees who would not otherwise agree to serve, likely would cause authors to avoid journals charging submission fees, and would require frequent currency exchange due to the international nature of science.

I consider the first three points to be logically flawed relative to the proposed system for the following reasons (in the same order as the original objections):

1. We’re all broke in PubCred land. We all start with zero credits and have to earn enough to submit manuscripts. If we replace credits with a fixed payment – fixed fee system where each reviewer is paid one third of the cost of a submission for each review then this is exactly the same situation as if I have $0 in my bank account. I have to review 3 papers to have earned enough money to submit one.
2. It doesn’t matter how high these numbers have to get because they are offset by the cost of submitting a paper.
3. Just like the proposed PubCred system, this only works if a large number of powerful journals are involved in a coordinated manner. Clearly having a small number of journals implement either system will lead to authors simply avoiding those journals (as happened to IEE when it tried implementing the money based system).

So, it seems to me that a simple logical substitution of dollars for credits negates all but one of the supposed objections to a monetarily based system. The final point related to currency exchange simply seems inconsequential.

In addition to being more straightforward than implementing a new PubCred system, I think that a monetary approach has an additional advantage. It allows the market to operate on the peer review system. I’m sure that I haven’t even begun to imagine all of the ways that the market could influence peer review, but here’s a short list of things that come to mind:

1. Great reviewers could be rewarded more than mediocre reviewers. PubCred treats reviews dichotomously. They are either good enough for credit or not. But we all know that reviews and reviewers don’t just fall into two groups, so why not reward reviewers on a sliding scale. Each journal can keep track of the quality of reviews and use that information to decide how much to offer a reviewer to entice them to review.
2. Journals that want papers reviewed faster can offer higher payments to entice reviewers.
3. Top journals that want more reviewers can charge higher submission fees to cover the expense.
4. Down the road this potentially provides an avenue for appropriately charging for-profit journals for the massive amount of… free… labor upon which they rely to make large profits.
5. Funding agencies and universities could potentially stop funding publication costs.

In conclusion I should say that I am super impressed with Fox & Petchey for being some of the first folks out there to actually put forth a serious suggestion for fixing the current problems with peer review and I think that they have (with an appropriately long lead-time and substantial up-front investment) come up with a system that would actual work. It’s just that overall it seems like there is a much simpler approach available. Take their approach, replace each credit with a fixed number of dollars (to start), and as a result get rid of all of the decision making and infrastructure.

UPDATE: Owen Petchey’s name is now spelled correctly. Sorry Owen.

——————————————————————————–

†Even the most basic rule of a 3:1 ratio of reviews to submissions seems like it should be a topic of discussion. What about journals like Science and Nature that due to an abundance of caution often get 3-5 reviews on a manuscript instead of the standard 2. Because the current proposal does not allow different journals to charge different numbers of credits for submissions or provide less credit for reviews, journals that utilize more reviewers will put a burden on the system (NB: editors also receive credit for managing manuscripts so the 3:1 ratio is really appropriate for a standard 2 reviewer system). So, should the ratio be increased to 4:1 or 5:1 or should journals be given flexibility with regards to credits and/or payments?

*We here at JE have noticed an increasing trend of late in the number of re-reviews requested and an apparent unwillingness on the part of some editors to take the time to evaluate whether the changes recommended by the reviewer have been provided. Instead they simply keep sending the paper back to the original reviewers until they have no comments left. This slows down the system, wastes reviewer time and motivation, frustrates authors, and under the proposed system there is no disincentive to stop editors from doing this ad nauseum – the reviewer has no recourse because if they don’t complete the potentially never ending re-reviews they receive no credit.

‡Who are in most cases motivated more by profit margins than the good of science. This is perfectly reasonable given that in the vast majority of cases they are private corporations, but it means that we don’t want them being the ones who are making critical decisions that would have large impacts on the scientific process.