The other day I had a little Twitter exchange with Andy Farke (of the Open Source Paleontologist) about the issues of finding referees for papers as an editor. Andy noted that there was not only a high refusal rate (people not wanting to review papers) and some referees being repeatedly nominated as choice targets by the authors of papers. I’ve not done that much as an editor and that’s probably why I’ve not seen as much of this as he has, but I can certainly see how it can be an issue.
Either as an author suggesting referees or referees picking one, there are lots of people to try and avoid. Clearly you’re not supposed to go for close collaborators or former students of the authors as they might be biased, and equally avoid people with an axe to grind (oddly many researchers don’t like you publishing papers that take down their pet hypotheses). You also need to try and pick people who provide good, fair, reviews and on time. I’ve catalogued so of my own travails with late referees before and it’s not a lot of fun to wait months and months for a reply only to get a few lines worth of comment.
Of course the referee also needs to be an expert in the area(s) concerned. It’s perhaps not a big surprise that this can prove tricky. By the time you’ve eliminated the referees that can’t or won’t review something, the ones that are always late, the nemesis of the lead author, his former students and best friends, and the ones you have asked 10 times already this year you can imagine the pool runs very shallow indeed. If that starting pool is small enough or has a lot of antagonists (he said while totally not thinking about pterosaurs at all) then it’s perhaps not a surprise that editors can struggle.
While the pool can’t easily be expanded it would appear that some people do need to be more willing to review at all, or on time if they do. I do know that some editors will keep a list of good and bad referees, but I wonder if any journals / editors offer feedback to referees (if they do I’ve never had any or heard of it). It’s odd, we go to a lot of trouble for authors to reply to and comment on the feedback they get from referees and argue things through, but why is less attention paid to the referees themselves? The can be every bit as influential on the work, and certainly I’ve come across reviews that paint the referee in far from a good light. Is it time to start handling and even reviewing the referee’s performances?