Posted by: cjlortie | March 19, 2012

May the odds be ever in your favor: a brief comment on the review games in ecology.

Is there a peer-review crisis in ecology? This is critical question that we need to explore to determine whether we should consider introducing changes. Referee selection, number of reviews needed, and the relative importance of referees versus editors in improving the quality of our manuscripts are viable lines of inquiry. There are numerous ways to tackle these questions. To assess the extent that we are in a crisis, I conducted an online survey to calculate the individual decline to review (weighted analysis of requests versus reviews actually done).

I am biased on this topic as I assumed there was a crisis. I also want ecologists to examine the why and how of reviews and this would be a useful indication that we need to so. So, a high decline to review rate would mean that most of us are too busy to review the work of others. And the answer is… in a short editorial on the topic published in Immediate Science Ecology.  It is the article at the bottom of the main page, and the pdf link is to the right of article.  The DOI is still pending, but the article is otherwise final.

In case you just want the punchline, the decline to review rate for ecologists is 49% (+/- 0.02) meaning that about half the requests are declined by the most appropriate referees. This is not a large pool of respondents nor necessarily indicative of all ecologists or what individual journals such as Oikos experience. Nonetheless, I see this a clear signal that we need to add incentives or change handling in some form.

Not included in the article was the importance of gender.  Analyses of productivity and role one serves in the process were however reported. Women accounted for 36% of the respondents, and interestingly, declined to review less often than the male respondents in this dataset. The decline to review rate of men was 1.5 time greater than women. There were differences in the proportions of respondents by gender that participated in peer review in various capacities with 20% of women serving as editors compared to 30% by men. Nonetheless, women do more reviews relative to the number of requests they receive (Figure below), and similar to the relationship reported in the editorial this difference increases with productivity.

So, this is likely a very real cost that women pay in science and is thus a compelling argument for ensuring that we consider managing the diversity of referees we choose to use in reviewing the work of others – not by gender but by career stage (junior versus senior) or by productivity in a specific field. After all, peer review is also an opportunity for collaboration and for building networks for practicing science. In summary, whether we agree on what constitutes a crisis or not, there is an opportunity here for journals in general, and for Oikos in particular, to consider how we select referees and whether it is meaningful to make these criteria transparent and ensure that reviews are spread between junior-senior, referee-editor, or highly productive veteran-newcomer to the games.

Gender redux, slight nuance in the race to publish. I was thinking about the comment posted by JEB on the meaning of the fit lines. Increasing productivity generates a greater divergence between requests and reviews likely due to a visibility effect, i.e. those that publish more papers also receive more solicitations.  Both genders demonstrate this effect, but I was wondering if it an’important’ difference (i.e. red versus blue line, like Tron light cycle races when they crash). I did a t-test for each gender to examine the scale of difference between the fit lines of requests-reviews (incidentally the r2 values are around 0.4). The two fit lines were significantly different for each gender – statistically speaking (t-test for females, t = -2.3, p = 0.02; t-test for males, t = -5.8, p = 0.0001), but the mean extent of divergence between the fit models is double that for men relative to women. This corresponds nicely to what your eye can see instantly in the plots that the lines are more different for men. However, the nuance I was considering is whether the female lines in predicted reviews catch up or overlap with males if the productivity scales fully overlapped, i.e. if additional female respondents completed the survey and reported more publications per year. They would not (t-test, t = -4, p = 0.0001) – women that publish more papers would still do more reviews. Fascinating! Disclaimer, this is only a dataset of 257, so I am purely speculating here.

Advertisements

Responses

  1. Chris – You cite your weighted decline rates, but present those regressions on ln(reviews or requests) ~ ln(pubs) – could you explain them in more detail, as they look interesting and appear to have some more juicy info in them?

  2. hi Jeremy,
    Yes, those values in the plots are unweighted just to keep it simple. I did the decline rates in general though on weighted values since if you got 2 requests total… there are not many options except 0,1,2. For the plots, I wanted to see if the requests exceeded the reviews and if there was evidence for a greater divergence between men and women. The d2rr rates are cool to show too but I plotted these up to match figure 2 in the paper.
    cheers,
    chris.

  3. I mean JEB. Sorry. I have been thinking on your comment further, and I will update the post in a second with a bit more interpretation.

  4. […] learning continued entirely due to posts and emails by readers. See below the movie poster in the post, I added a section of additional analyses and interpretation generated by discussion with […]

  5. […] and blogger Chris Lortie has a strong interest in scientific publication practices (see, e.g., here). His latest effort, now in press at Oikos, examines patterns of funding and impact among the […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: