Posted by: cjlortie | December 28, 2011

Paid peer reviews for ecologists

About 6 weeks ago, I was invited to write an editorial for Ideas in Ecology and Evolution (IEE). I recalled reading a few articles by Hochberg commenting on the increasing difficulty in securing referees and had seen several new incentive systems including PubCreds discussed by oikosjeremy. So, I am just posting it here, not to promote it, but to provide a forum for discussion and synthesis on the topic.

The editorial summarizing the outcome of a poll I conducted on the ecolog listserve and my musing is online, open-access here (just scroll down a bit on that first page and you will see the current articles).  Click on the money for nothing one then on pdf and bingo.

http://library.queensu.ca/ojs/index.php/IEE/index

I would love to hear what the Oikos and Nordic Ecological Society thinks about this topic. I did this mainly as an exercise for us to push for more transparency in peer review in general (including the incentives and profit) and for us to consider whether the model we use to handle papers could be improved. Maybe the new EiC for Oikos will solve this.  I would also love to hear from the Oikos editors and referees to see if we would ever consider this.

cheers,
chris.

Advertisements

Responses

  1. The idea of paying referees has been discussed in various venues. I exchanged some ideas with fellow ecological blogger Ethan White a while back. See http://www.ipetitions.com/petition/fix-peer-review/blog/2680.

    Basically, the reason Owen Petchey and I thought up PubCreds was to get around various problems with paying peer reviewers with real money. One can of course argue about whether PubCreds has its own problems which are even more serious.

    • Agreed. Cool idea. It would be nice to be direct though. What are your thoughts on this now? Free page charges or colour reproductions next time you publish with a journal that you review for a lot (on time)?

      Or fast-tracked reviews of your own work? Like a coffee shop stamp card. Do 10 reviews, next paper submitted gets faster handling (no difference in outcome, just editors make a point to get it to referees much quicker).

      • The thing about those sorts of incentives is that they are hard to callibrate, and people place very different marginal values on them. A virtue of pubcreds is that it directly links your reviewing and submitting.

  2. what do you think is the best solution? I was thinking something like a coffee shop stamp card with each journal you review for. Review a lot, on time, and get free page charges or colour reproductions.

  3. I think it’s a discussion that’s well worth having, but as Jeremy points out, there’s usually a lot more to consider once you start getting under the surface of the discussion.

    If you’re talking purely financial recompense, one has to ask where the dough comes from? For-profit publishing houses might not be so keen to shell out cash to reviewers (even though their profits rely on our work). Society run journals may not even have money to spend on these things. Journals offering “fast-track” reviews can too easily be let down by reviewers, without offering extra incentives to entice reviewers to submit rapidly. And it’s arguable that we should be spending more time reviewing articles to improve quality, not less.

    The PubCred and related ideas get round that quite easily by dealing with other forms of recompense, but (as Jeremy also points out) there are other issues that need to be dealt with here, such as the “guns-for-hire” question of who deserves an authorship on a paper.

    The Scholarly Kitchen blog touches on many of these and related points (if you dig through their archives), as well as linking to those few academic studies that have actually tried to quantify the size of the reviewing problem – if it even exists. The evidence (scientific, rather than anecdotal) seems to suggest that the problem isn’t so big after all – in certain fields. This doesn’t include the fascinating suggestion that Quality reviewing declines with experience though.

    Personally, I think it would be great to have more academic credit for the reviewing work we do, as well as more transparency surrounding all of these issues. Plenty of interesting topics for discussion!

    • Thanks Mike. I will take a look at those sites. Academic credit is a nice distinction and really important. I am also familiar with the quality issue and the research on it too. Surprising.

      I was also musing on credit in that most academics have merit awards, tenure hoops to hop through, etc. so credit in the form of recognition would be nice to be able to bring back to one’s employer (if needed/desired). A standard form letter drafted by the editors that they provide at certain benchmark review levels could be cool. Like a frequent flyer rewards program card. You could frame it and put up on the wall even. ha.

      The other limitation I was considering was that you may review a lot with one journal but publish with another. So, society level credit would be nice transcending the specific journal. The other issue might be that you might not get asked frequently enough to earn credit. I suspect some established folks get many many requests whereas others get less. So, that is a bit tricky too.

      With respect to the crisis, that is the million dollar question! I would love to know how many individuals editors have to ask to get just two!!

  4. […] on from previous posts on reforming peer review (see here, here, and here), I wanted to note a new peer review service, Peerage of Science. PoS is a private […]

  5. Dear all,

    Put it like that: I mainly due reviews of papers during weekends or late in the evening because I am busy doing research, teaching, and administration for my institute during work hours. So, I am either getting less sleep or less time with my family to keep the system working. But the system is not run by colleagues. It is run by private corporations that make substantial profit out of a highly subsidized business and free work from a specialized labour force. So, I am sorry, but I don’t see a reason why referes should not be financially compensated for their work.

    The publishing business has to internalize its costs if it wants to be sustainable. Paying referees (and editors) is part of the cost of running such a business. Exceptions can be made for non-profit organizations, like professional societies, since members may want to contribute with their work for the development of the society. But I don’t see a reason why such logic would apply to the highly profitable business of Wiley, Elsevier, or Springer.

    Miguel

  6. Hello. I know I am late but I ran a search and this page came up.

    I could not find why paying with actual money was dismissed. Now that Impact Factors are losing their meaning, journals might start to use their reviewer stable as a measure of their prestige. The “bigger” journals would pay more and get bigger names to work as reviewers. We all do it for free now; I am sure we would be more willing to do it for 10, 50 or 100 dollars/Euros. The journals would, of course, be more careful about sending out papers for review. Every paper send out would be an investment, and every paper rejected would be a lost investment.

    I hope that I resurrect the discussion a little. Thanks for any replies.

    • Hi George,

      I’d suggest one reason that direct cash incentives might not be popular in the medium to longer term is the shift to Open Access (author pays) publishing models.

      In general, academics are looking for greater transparency from publication costs, and I would worry that adding reviewer financial costs would not necessarily clarify things. I think a few of the points I made above (where does the extra money required come from, how do we know who are the best reviewers in order to select and pay them appropriately?) would still be problems with direct cash payments to reviewers.

      This doesn’t mean a workable cash based system can’t be developed (e.g., you could discount reviewing cash/credits earned from the cost of accepted papers in an author pays journal/publisher), simply that it would probably be at least as messy as the current “review for the common good” approach.

      • I think the “paid reviewer” discussion is based on the assumption that the author pays. I was not suggesting that. I was suggesting that the journal pays. Where does the money come from? Well, from their revenues. Unless working for non-profit, charitable organizations, all other professionals charge for their services. Why not us?

        When they have to pay, journals editors would probably not send every paper for review, but that is okay. It would save a lot of unnecessary review work that now is done for free.

        It does not have to be much, just an “honorarium” at the start, but once the system get established, different journals would start offering different amounts. The top dogs (or the really busy people) would end up reviewing only for the journals that pay the most, presumably the most profitable and the “best” (yes, people will argue here, but the free market does work). The rest of us might get $5 per review and be happy with that. Now we do it for free.

        A problem here is the bundling of journal subscriptions, whereby to get journal A, you have to get the A through Z, but the free market will take care of that.

        Getting paid for reviewing removes a huge externality from the system, that is all. it would force publishers to come up with innovative ways of making profit.

      • Agreed! and provide better service to science and authors!

      • I think the “paid reviewer” discussion is based on the assumption that the author pays. I was not suggesting that. I was suggesting that the journal pays. Where does the money come from? Well, from their revenues.

        Journals will not pay for anything. They will simply increase the charges made to academics. This will happen directly through the “Author Pays” model, or indirectly through inscreased library/individual subscription prices through the “Reader pays” model.

        So far, the free-market has worked strongly in favour of journal profits at the expense of academics (Elsevier reportedly make a 30% profit on their academics journals). I´m not sure I would trust it to take care of all these important things so readily.

        Not-for-profit publishers, like PLoS, rely on other funding streams, like endowments and subsidising top rank journals (PLoS Biology) with the extra revenue generated from high-volume journals (PLoS One). As they don´t make a profit, they would need to increase article publication prices to cover the cost of paying reviewers.

        Having worked as an Editor on a similarly sized journal to IEE, I´m not sure it´s a good point of comparison. The Editorial workload is light, based on the low rates of submission, and can be covered by a few eager volunteers. Not at all comparable to Oikos, Nature or Science.

        There´s also a large paradox inherent in the system. Poor science will cost more to publish, as it needs to be reviewed more often. Not necessarily a good return on our limited academic funding.

      • Good points. They will stick to it us either way. Or more delicately, the costs will simply be redirected right back at us to protect their profits. Perhaps we need a more dramatic solution then, one that decouples the accessibility and feedback of our work from profits derived by others?

  7. Hi George & Mike,
    Even with open access, author-directed peer review (ADPR) as Lonnie Aarssen proposed for the open-access journal IEE is viable. I think compensation in some form is pretty useful, maybe not critical, but useful. Publishers make a lot of money under either model, open or closed, so paying referees is viable under both. Author pays to get paper published for open access still means money moving to publisher that in theory could reward referees.

  8. It seems peculiar that reviewing would be one of the few services that has to be provided free of charge. I can think of only one other service for which many jurisdictions prohibit charging a fee, but in places like Holland everything seems to work just fine.

    There is a third alternative to reader pays and author pays: advertiser pays. It seems journals have gone digital but their business model is still based on the old paper system. My google email account seems to be free, but it is not, it is paid for by advertisers whose ads pop up at the edge every time I read my email. Most of the time I ignore the ads, but occasionally I have clicked on one, and if I have more money to spare, I would probably have spent some of it. The ads are surprisingly relevant.

    Targeted ads could work. Every time you click on a journal page, a given issue, or a given paper, the ads could be more finely targeted. Once you download the pdf the ads would end, so your paper would not be specifically sponsored by anyone or support anyone. One would have to establish a strong ad/science barrier, something similar to the blood-brain barrier, which supports the brain but prevent any negative effects. That is the benefit of “per click” systems. the advertisers does not even have to know what the journal, issue or paper is about. It is all done automatically. As long as people continue clicking, it does not matter, for either party.

    The numbers might not balance completely at first, but imagine,, with this system, the most successful journals might end up being completely free for both readers and authors. Maybe we can convince someone at google to start a journal with a new business model. They claim to be interested in the free dissemination of knowledge.

    • Awesome idea. We could also filter the providers to ensure it is appropriate and ethical content/services being advertised.

      • As I understand the google system, the ads’ ranking depends on peoples’ clicking, so if nobody clicks on an ad, the ad loses its ranking and disappears from view. There is no need for censoring.

      • Aha, that is good. It could still be visually offensive though 🙂

  9. Call me a liberal, but I prefer no censorship. We are all adults. In any case, advertisers are not in the business of wasting money that is the benefit of targeted advertising. Nevertheless, we stray from the main point.

    • It’s a nice ideal, George, one that I wish were true, but we have too many examples to the contrary to believe that scientists and/or academic publishers are more moral or ethical than the general public. To maintain the strength and integrity of our important messages, we have to ensure that accusations of conflicts of interest are kept at a minimum (whether well founded or not) are kept at a minimum.

  10. Fine. You can be the censors.

    • I get the collective commons so to speak would easily take care of all this with no censorship needed. Reputation is usually incentive enough to be good.

      • It is amazing how a discussion about new models of reviewing and publishing degrades into a discussion about censorship.

      • This always happens since we have so many gatekeepers in science and science it is sadly about control.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: