Posted by: cjlortie | January 5, 2012

Is there a referee crisis in ecology?

Dear Oikos and Nordic folks,

Thank you so much for your feedback on the editorial ‘Money for nothing and referees for free’ published in Ideas in Ecology and Evolution in December. The most compelling and common question I was asked was is there a referee crisis in ecology (or tragedy of the ‘reviewers common’ as Hochberg et al. proposed). This is an excellent question. I propose that whilst there are more perfect ways to test this (total up # of submissions and then estimate total pool of referees, tricky), an interesting indicator would instead to be calculate the decline to review rate (d2rr) in ecology. I envision the following two primary data streams to calculate this rate: a per capita estimate derived from each of us personally and a mean estimate of rate from the publishing portals (journals). Hence, let’s do it. Only you know your decline to (accept doing a) review rate across all requests whilst journals track their own net rates and your specific rate with them too.

So, please take 30 seconds and fill in this short survey, and we can then assess to an extent whether there is a referee crisis in ecology.

https://www.surveymonkey.com/s/VD3K36W

I have also compiled a long list of emails for every editor I could find for all ecology journals and have contacted them to see if they would share the rate at which individuals decline for each of them, i.e. do they have to ask 5 or 6 people to even secure two reviews? I will not share the journal names etc. and protect their rates as I recognize the implications. I would just like to know what our overall mean is from a journal perspective too.

Thanks so much for your time and help with these discussions. I hope you think they are important too, but I also want to assure you that this is my penultimate post on the subject.

Advertisements

Responses

  1. Hi Chris,

    Owen Petchey and I tried to poll journals for similar data (and other lines of evidence as well), to look for various “symptoms” of a “tragedy of the reviewer commons”. While we mostly got positive responses, very few journals followed up and actually provided data, with the exception of the BES journals (whose data Owen and I are now currently analyzing, but which we are not authorized to share). For discussion of our efforts on this front, see

    http://www.ipetitions.com/petition/fix-peer-review/blog/2983
    http://www.ipetitions.com/petition/fix-peer-review/blog/5040

    And see this post

    http://www.ipetitions.com/petition/fix-peer-review/blog/3313

    for discussion and links relating to a couple of big international ‘snapshot’ surveys of the state of the entire academic peer review system. These surveys have the disadvantage of not providing time series data, and being very coarse (they’re about all academic research, not a specific field), but they are real data rather than anecdotes and the sample size is pretty large. Overall, they paint (in my view) a picture of a system that is under strain and potentially vulnerable, but not yet in the midst of a full-on crisis. The surveys also asked questions about potential remedies, such as various sorts of incentives for review.

    At Oikos, it is worth noting that, back in 2006 when I first joined the board, Linus used to ask editors to suggest 3 referees in order to be reasonably sure of getting 2 reviews. But a few years ago, he started asking the editors to suggest 5 referees to be reasonably sure of getting 2 reviews. This suggests that finding referees is more difficult than it used to be, although of course it’s only for one journal and it doesn’t indicate a crisis.

    Whether we should wait until a crisis hits in order to deal with it is another question. Perhaps not, if the crisis is predictable. I have yet to see anyone claim that the incentives to review are well-matched to the incentives to submit. Academics must publish or perish, not review or perish. Indeed, everyone seems to agree that the incentives are only becoming more mismatched over time. That would suggest that a crisis is to be expected at some point. So it’s at least worth thinking about how to address or prevent it before it arrives, even if we don’t know with any precision when it will arrive, and even if preventive or corrective measures would be costly.

    • Cool. Thx Jeremy. I will you posted too and certainly review all those sources you listed. Wow, thanks.

    • Also, I suspect the crisis is already upon us. Perhaps time for alternative models to publish.

  2. Are folks likely to respond to the survey more likely to accept reviews? 🙂

    • Yes.

      • Could you include something that accounts for, on average, how many times a paper has been submitted? For example, someone could publish 5 papers in a given year, but those may have been submitted to different journals several times, so the total number of submissions may be far, far higher. I’d guess it’s at least twice as high (e.g., Cassey & Blackburn 2004 BioScience). This seems important if you’re weighing submissions versus reviews.

  3. Yes, this is a great idea too. I have heard that some journals now ask if they ms has been reviewed previously at another journal. The total ‘real’ workload of the community must be much greater. I will see what the journal level dataset provides too.

  4. Dear colleagues,
    I have been an editor for some journals which also publish ecological papers (although they are not only ecological): Amphibia-Reptilia, Chelonian Conservation and Biology, and Endangered Species Research. I have been co-editor of the first of these journals for eight journals, and I also noticed that finding referees became much more difficult in the last years. Cannot quantify exactly, but the trend was clear, and it is the same for the other two journals. I suppose the crisis is general in science, and not only in ecology (referees are increasingly difficult to find also for taxonomy, physiology, etc papers). I suspect that the huge increase in the number of journals and papers submitted, is the core of the whole problem (everybody of us receive too many papers; for instance I received nearly 40 for review in the 2011); so I don’t think we can solve this problem unless totally different peer-review procedures will be adopted.
    Luca Luiselli

  5. Hi Luca,

    The anecdotal comments are all to this effect from almost everyone. So, I am thinking too that rewards or compensation of any sort won’t do the trick but that we adopt a very different system altogether. We have discussed compensation, not financial though, at Oikos but not a different system. Dr. Aarssen at IEE tried author directed peer review (ie authors provide their own reviews) but people were hesistant. Maybe we now have to consider options like this more seriously though. Have your journals come up with any solutions?
    cheers,
    chris.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: