Referee selection roulette

The other day I had a little Twitter exchange with Andy Farke (of the Open Source Paleontologist) about the issues of finding referees for papers as an editor. Andy noted that there was not only a high refusal rate (people not wanting to review papers) and some referees being repeatedly nominated as choice targets by the authors of papers. I’ve not done that much as an editor and that’s probably why I’ve not seen as much of this as he has, but I can certainly see how it can be an issue.

Either as an author suggesting referees or referees picking one, there are lots of people to try and avoid. Clearly you’re not supposed to go for close collaborators or former students of the authors as they might be biased, and equally avoid people with an axe to grind (oddly many researchers don’t like you publishing papers that take down their pet hypotheses). You also need to try and pick people who provide good, fair, reviews and on time. I’ve catalogued so of my own travails with late referees before and it’s not a lot of fun to wait months and months for a reply only to get a few lines worth of comment.

Of course the referee also needs to be an expert in the area(s) concerned. It’s perhaps not a big surprise that this can prove tricky. By the time you’ve eliminated the referees that can’t or won’t review something, the ones that are always late, the nemesis of the lead author, his former students and best friends, and the ones you have asked 10 times already this year you can imagine the pool runs very shallow indeed. If that starting pool is small enough or has a lot of antagonists (he said while totally not thinking about pterosaurs at all) then it’s perhaps not a surprise that editors can struggle.

While the pool can’t easily be expanded it would appear that some people do need to be more willing to review at all, or on time if they do. I do know that some editors will keep a list of good and bad referees, but I wonder if any journals / editors offer feedback to referees (if they do I’ve never had any or heard of it). It’s odd, we go to a lot of trouble for authors to reply to and comment on the feedback they get from referees and argue things through, but why is less attention paid to the referees themselves? The can be every bit as influential on the work, and certainly I’ve come across reviews that paint the referee in far from a good light. Is it time to start handling and even reviewing the referee’s performances?

5 Responses to “Referee selection roulette”

  1. 1 Heinrich Mallison 23/03/2012 at 9:10 am

    That’s a bad enough situation for a paper. Worse, though, is that grants applications are equally affected. Imagine waiting for an extra 6 month, on the dole, because the funding agency can’t find reviewers! If you really specialize in a narrow, complex and interdisciplinary area, the pool of capable reviewers may be two deep only – both on the application as well. Oops!

    [sarcasm]not, I am NOT speaking from personal experience, not at all. [/sarcasm]

  2. 3 Mickey Mortimer 23/03/2012 at 9:18 pm

    How would people feel about a public database of reviewer quality, like the Rate My Professors site? I understand that reviewing papers is usually considered part of an academic’s duty, so it seems only fair that their quality in this regard be made public for those who require their services. This would also enable social pressure to help scientists contribute to the community, and would be an easy way to see who boycotts reviewing for groups like Elsevier. It would save authors time and effort, and could be incredibly useful in the goal of not depending on publishers.

    • 4 David Hone 24/03/2012 at 8:10 am

      Anything that encourages people to do a better job is a good thing, but I’d imagine it would be strongly resisted by those who don’t do reviews.

      One suggestion I heard was for referees’ names to appear on papers. The paper could still be reviewed blind, but at the end it would say somethign like: Referee 1 DWE Hone, 1st review, minor corrections, 2nd review, publish. Referee 2. JOHN Doe, 1st review major corrections, 2nd review, publish.

      The idea being that it would be obvious in a lot of cases where people had passed off terrible papers with good reviews, or terrible papers were published despite scathing reviews. It would need fine tuning, but as a concept I think it’s quite a good one. But then perhaps I would because I tend to review lots of things, on time, and try to be fair and only very rarely hide behind anonyminity….

      • 5 Mickey Mortimer 24/03/2012 at 8:20 pm

        Well, the people who don’t do reviews wouldn’t have any say in the matter, just like the crappy professors didn’t get to vote whether Rate My Professors was created. And while they might be annoyed that their habits are being made public, on the plus side from their perspective, at least they would get less requests for reviews since everyone would know they don’t bother or do a poor job. Then again, the liklihood of anyone else reviewing their papers is now lowered, but how could they argue that was unfair?

        I agree with your concept of less anonymity. Though again, without making review recommendations public, that just means everyone has to waste time going through the same motions to find out who the poor reviewers are.

Comments are currently closed.

@Dave_Hone on Twitter

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 453 other followers

%d bloggers like this: