Posts Tagged 'peer review'

Referee selection roulette

The other day I had a little Twitter exchange with Andy Farke (of the Open Source Paleontologist) about the issues of finding referees for papers as an editor. Andy noted that there was not only a high refusal rate (people not wanting to review papers) and some referees being repeatedly nominated as choice targets by the authors of papers. I’ve not done that much as an editor and that’s probably why I’ve not seen as much of this as he has, but I can certainly see how it can be an issue.

Either as an author suggesting referees or referees picking one, there are lots of people to try and avoid. Clearly you’re not supposed to go for close collaborators or former students of the authors as they might be biased, and equally avoid people with an axe to grind (oddly many researchers don’t like you publishing papers that take down their pet hypotheses). You also need to try and pick people who provide good, fair, reviews and on time. I’ve catalogued so of my own travails with late referees before and it’s not a lot of fun to wait months and months for a reply only to get a few lines worth of comment.

Of course the referee also needs to be an expert in the area(s) concerned. It’s perhaps not a big surprise that this can prove tricky. By the time you’ve eliminated the referees that can’t or won’t review something, the ones that are always late, the nemesis of the lead author, his former students and best friends, and the ones you have asked 10 times already this year you can imagine the pool runs very shallow indeed. If that starting pool is small enough or has a lot of antagonists (he said while totally not thinking about pterosaurs at all) then it’s perhaps not a surprise that editors can struggle.

While the pool can’t easily be expanded it would appear that some people do need to be more willing to review at all, or on time if they do. I do know that some editors will keep a list of good and bad referees, but I wonder if any journals / editors offer feedback to referees (if they do I’ve never had any or heard of it). It’s odd, we go to a lot of trouble for authors to reply to and comment on the feedback they get from referees and argue things through, but why is less attention paid to the referees themselves? The can be every bit as influential on the work, and certainly I’ve come across reviews that paint the referee in far from a good light. Is it time to start handling and even reviewing the referee’s performances?

Where are my papers?

Let’s face it, it’s been a while since I had a good complain about something, so in the usual holiday manner (the spirit is  supposed to be merriment, but let’s face it, the tradition isn’t!) here’s someone grumpy complaining. There’s another moan to follow tomorrow but I’ll sweetn the deal by following this with comments on theropod sociality and my reviews on the zoo and aviary in Pittsburgh.

………………………….

It’s customary for me to whine about reviewers and editors periodically and for once it has been a while since my last effort. However, the Christmas break has allowed me to try and catch up with a few little things, one of which has been to see what has befallen various papers I’m involved in and if there has been any news of them. While I do have a pretty large volume of manuscripts with various journals, to be honest it’s not pretty reading. Now sure there are valid reasons for papers being delayed (and of course the Christmas period doesn’t help), but you would hope that the occasional paper would run to time, or be processed in a timely manner.

By my count I currently have 9 manuscripts in out with journals. Of these, based on the ideal review times listed by the journals or what I can remember when being asked to review for them, 8 are now late. The last one will be late if it’s not back to me this week and I have good reason to think it won’t be.

There are couple more which have recently been returned, one in a timely manner and one late, and there are a couple of book chapters which are literally years overdue. More than that, despite contacting editors about them, in some cases I have no news at all what is happening to the paper (including one submitted in July!) and in one case the manuscript is awaiting assignment of referees when it is a resubmission. You’d assume they’d be sending the paper back to the same people, and even if they refuse to review something a second time, does it really take three months to send them out?

In short my manuscripts are late by the standards of the journals themselves. In more than one case things are profoundly late, and in a couple I can’t even find out what has happened to the manuscript. Referees seem to run late as a near matter of course and often they are given months toe review something a handful of pages long anyway, something that annoys me profoundly. But when papers aren’t even being sent to referees for weeks, even months, it’s very annoying. Even if a referee is superb and turns around a review in a few days, if it didn’t reach him for weeks, or the review doesn’t reach me for weeks, then the whole thing is going to be late. It’s especially when journals try all these little tricks like publish uncorrected proofs and the like to get the papers out as early as possible. So, it’s clear they value a paper that’s ready for publication being hurried into availability, but then they make no effort at all to actually have papers edited or refereed in a timely manner.

Now sure, maybe this is happening to everyone, but really is that an excuse? As I’m fond of saying about this, writing that review, or mailing out to ask for referees, or check a set of corrections or whatever takes the same amount of time to do today as it will in 2 weeks, or 6 weeks or even 6 months. And while you might be busy this week, and even next, I don’t think it’s excusable to sit on something for months at a time. It does the author a disservice and for that matter both the journal and the field as a whole. Science is not served by papers, perhaps important papers, being held up by months, even years, because people won’t do the work they said they would.

Is it really this bad for palaeo, or am I profoundly unlucky? Looking back over my past papers and various submissions I would say the average review time for a manuscript of mine is about 5 months, and I’ve had half a dozen that were over 6 months from submission to return. Conversations with colleagues suggests that I have had some bad luck and the extremes I’ve occasionally suffered (over a year on 2 occasions, and several more over 6 months each) are the exception. Even so, I’d be intrigued to know what this is like for my colleagues and indeed for those in other fields of science and research.

“Any jackass can trash a manuscript….”

“….but it takes good scholarship to create one.”

This title is taken from an editorial in Molecular Biology in the Cell which basically admonishes poor refereeing techniques and lays out how these should be done and how both editors and authors should deal with them. It’s all very, very good stuff, I just hope people pay attention to it. If only more people or journals went with this kind of thing.

It’s always nice when you shout about various things quite loudly (like this, this and this) and then see that quite independently in a rather branch of study the same problems are being felt and the similar solutions proposed.

Thanks to Graeme Lloyd for putting me onto it.

Playing the game

I’d been thinking of writing this post when I discovered that the theme I had intended to expand upon was rumbling on itself over on SV-POW. That theme is on amateurs publishing in palaeontology. Yesterday I moaned about poor papers and the effect they can have by generating huge amounts of awkward work for researchers to set the record straight. Here I want to make the point about why, or perhaps how, this happens.

Science and scientific research is open to everyone. It is supposed to be open to everyone, if you have an idea or some research and can back it up with evidence, you should be able to demonstrate it in some form of public forum. Now researchers prefer peer-reviwed journals as this does result in some basic weeding out of poor practice (mistakes in methods, data collection etc. etc.) and ensures that people are likely to find your work and able to publish replies and so on. However, any book, magazine or journal should be OK in theory.

In practice it is another matter of course, because of this very openness. By opening ourselves to any work of any form then we risk the presence of poor work: inclusiveness at the expense of quality control. The simple fact is that we want everyone to participate in the game of research we really don’t set rules and simply expect (or perhaps rather, hope) that people follow the protocols that most of us do. It is those who want to play the game, and indeed are joining in, that have not bothered to learn the rules, or don’t know they are there, or deliberately snub them, that cause the problem. Since we don’t enforce those rules by say deliberately setting prescedent that only peer-review counts, or can’t in the private games outside of peer-reviewed journals, then all manner of problems surface and are allowed to surface.

Ignorance can always be forgiven, but it generally takes at least some decent form of basic knowledge of the process of both science and scientific publication to even try. Thus one cannot help but suspect that in many cases, perhaps most, or even the great majority, the reason the rules are circumvented is a combination of laziness, overconfidence or deceit. Such poor papers are being produced by those who want to take part in the game, to be seen to be taking part in the game, but are simply not playing by the rules.

Unless science makes a concerted effort to shift the rules or enforce them in a different way (and I think it neither will, nor should) we will have to put up with these issues. But it would help a great deal if these people would both realise what they are doing and the problems they are causing. The game would be more enjoyable and go faster without these intrusions and that should be possible. It is great that anyone, everyone, can get involved in real science, but there is a significant difference between doing it right, and just doing it. Sadly there are far too many people who seem not to get this, but one hopes things will improve.


@Dave_Hone on Twitter

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 502 other followers