Problems with peer review

There have been a number of posts both here and elsewhere on the mechanism of peer review (how papers are refereed before publication) and various comments coming through on the subject. Based on these and my own discussions with various colleagues as well as my experiences as a referee, editor and author, it seems to me that there are significant problems with the system. (I could have just been unlucky, or it might be really bad for dinosaur papers or palaeo ones, but discussions with colleagues in other disciplines suggest otherwise). However, I would argue that some of them could be alleviated or even fixed with a few relatively simple measures and others could be improved dramatically. Ultimately the real problem I suspect is simply a lack of time and or / funds to do things better and this quite possibly will never change. However, if sufficient people make sufficient noise about a problem, change can happen. This post therefore is going to consist of me putting forward what I think are serious problems with peer review, and ways that these can be improved. Everything is hypothetical (I can hardly be sure they will work, even if they are ever tried) but I think reasonable. I’d be very interested to hear what other professionals think of peer review as it stands and what could or should be done to improve the situation. Obviously in principle, the idea is fine, and many papers are reviewed and refereed properly and correctly, but this is not always the case (indeed could be argued is rarely the case) and this *is* a problem.

Before you start, if you do not know much about peer review and / or have never tried to get a scientific paper published I suggest you read my guides to review and publication (and some of the comments too) as a starting point as to the process that this is about. Right onto the list, which is in no particular order:

1. Proper separation of responsibilities of the editors and referees. It seems that referees are often doing the editors job of checking things like formatting and there is general toe treading which causes confusion and duplication of work. This is related to:

2. Insufficient guidelines for referees. Referees are often just given a paper and told to review it. Even when there is some kind of form to fill in, I have yet to find one especially helpful as I am often faced with yes/no choices for answers when I want to write a sentence or two to explain my decision or add a qualifier. An example was a review I did for a dedicated volume as part of a wider series. The question asked if I felt the paper would be of general interest – to which of course the answer was ‘no’, it was intended for a specialist volume, but then an unthinking (or unaware) editor might see this as a serious negative point against the manuscript. Referees need clear guidelines as to what they should and should not comment on, and be given the freedom to express those opinions even within a framework. Given just how many papers get refereed for the average journal each year, this should not be too much to ask.

3. Poor referee selection. Authors are often asked to submit a list of potential referees for their papers yet in my experience these are rarely selected. This means that I am wasting my time writing out lists of researchers with contact details, and appropriated referees are not being selected. Of course not everyone can referee every paper and editors are limited in their choices, but often entirely unsuitable referees are chosen to review papers. If your paper is based on a cladistic analysis, it should not be refereed by someone who does not use cladistics. If you paper is on dinosaurs, it should not be reviewed by a mammal specialist. It is incredibly frustrating to have a paper rejected when it seems clear that the referee simply did not understand or appreciate what was being done. If you write a paper on dinosaur behaviour, yes you probably want a dinosaur expert as a referee to check the palaeo side of things, but you also need a behavioural specialist to check the ethology. In practice, a palaeo editor at a palaeo journal will pick two palaeontologists since they don’t know of any appropriate referees. It might be beyond reason, but why can’t we have a database of researchers and what things we can (and cannot referee) for editors to check. Referees must also take responsibility here and refuse to review papers for which they are un- or underqualified to review, and demand that appropriate referees are chosen. If not, good papers are rejected because they are not reviewed properly, and bad ones are published for the same reason.

4. Incredibly poor time keeping. I recently had a paper that took eleven months get a response. What was more galling was that the editor apparently had it for six months before he even sent it out for review. Referees often take six months to a year to review a paper. This is unacceptable. Yes referees and editors are busy, and they don’t always have the time to deal with something instantly, it takes time for an editor to find referees willing to do the work and time to sit and write a review. However this is stupid. If a review takes four hours to write it will take four hours to do today or four hours to do two years from now. Editors cannot be allowed to take months to send papers out for review and referees cannot be allowed to take months to write reviews. This is hard to fix since the motivation must come from the journals and editors / referees themselves and I am not sure what this motivation could take. My guess is that a journal that got a reputation for rapid and good refereeing, more people would send their manuscripts there which would improve the quality of the work going into the journal, improve the journal’s quality and popularity and benefit everyone (Nature manages to get review back in two weeks, why can it take more than a year for others?). At the least, the authors need a right to reply to editors or senior editors about review times. This leads to:

5. Improved dialogue with authors. Authors (especially young and inexperienced ones) are unwilling to challenge reviews or question editors over decisions. This is understandable (I would not want to complain about incompetent or unfair reviews to an editor who might review my next paper, or be looking for a student or reviewing my next grant) but also unfortunate. At the least this must be a fundamental part of submitting a paper – the author must be told how to contact the chief editor, or handling editor and offered advice on how to raise questions / file complaints. I am sure editors would hate it, but problems would be far better resolved with far fewer hurt feelings if there was a clearer way to resolve conflicts over publications. A more general dialogue between authors and editors would also help cut down on delays and hopefully force editors to get things done to a schedule and deal with referees better.

6. Tendency to focus on the negative. This might sound silly, but often reviewing forms when sent out are framed such that referees are expected or asked only to note problems with the paper in hand. This can place an undue emphasis on the negative (“what problems must be changed?” vs “what are the strengths and weakness of the paper?”). I have had feedback from referees of papers that were rejected who though the actual work was good and publishable, but since they only send in a list of problems / corrections, the editors did not like it. (This is of course not the only reason, but you take the point I hope). Obviously the reviews will always look at what is wrong, since you don’t need to change or comment on what is good or right, but it would be good if referees in general, and editors / review forms specifically encouraged, people to provide a short list of major strengths or good parts of the paper. This will help the editors make a more balanced decision, and help the authors learn what is considered to be good science, not just what is bad science.

7. Moderating editors’ responses. This harks back to the communication issues and the ability to challenge or at least, request explanations for, decisions. Obviously editors have restrictions and limitations on what can go into a journal – even excellent papers may be rejected if there is simply no room, or the focus of the paper is not of interest to the audience they are targeting. However, I have certainly had judgments from editors that strongly suggest that they have read neither the paper in detail, nor the comments of the referees. If the referees’ opinions strongly conflict the editor must make a balanced judgment, or better yet go to a third referee to get a further opinion. It is frustrating to get a terrible review where you feel the referee has completely missed the point of the paper, claims you have done things you have not, or not done things you have and then find the editor things this is entirely reasonable. If the editor is unsure if a paper is suitable for publication, I am not sure what the harm would be to speak to the author and ask their opinion. They can discuss the issues with the paper and if it can be redirected a little to get it published, I doubt many would be unhappy with this, instead of just getting a flat rejection. The editors can also speak again to the referees, I recently recommended a paper for rejection and later saw it published with barely a change made to it – the editor is of course happy to disagree with my verdict, but I felt and stated that there were a number of basic and indeed critical errors to the paper. I can’t help but think that a better dialogue would have helped get these points across and a poor paper could have been made (I think) into a decent paper.

8. Uninformative reviews. There is no point in rejecting a paper as a referee if you can offer no more than ‘reject this paper’. Editors should *reject* reviews that are short or uninformative. First of all they imply that the paper was not read in detail (which if the case is both a disservice to the author and if the paper is rejected as a result, also unfair) and secondly it does nothing to improve the paper. I have recommended outright rejection for papers that were wrong from the title to the appendices but still offered a few pages of what I hoped were constructive criticism as to how the authors might approach a second approach to the problem even if I thought nothing was salvageable of the manuscript at all. Yes it takes time, but it means that next time the author might write something better, they should have learned something and on average the amount of time wasted will go down if fewer bad papers are submitted and the quality of the science will go up.

All of these issues I can relate to personal experiences with papers I have worked on, or heard from close colleagues and collaborators. Discussions with others suggest that these are far from unique and are frustrations experienced by all ranks in the profession, and not just young researchers (though I think we tend to be the worst affected as noted above), or limited to palaeontology, or even dinosaur and pterosaur research. These are important problems since science (and academia) as a whole depends on the peer review system and if good papers are being rejected and poor ones published this is bad. If papers are being held up unnecessarily for months or years though slow refereeing or incorrect rejections forcing resubmissions this is bad. If people are not learning to improve their work though not getting suitable feedback this is bad. And if referees are reviewing papers they are not qualified to process or are wasting their time by duplicating the work of other this is bad. None of these problems are insurmountable, or even complex, they just require some thought and application.

If reviews are carried out better, then better research will be published. If referees provide more detailed descriptions and better feedback, then researchers will learn to produce better work more quickly. This will produce both better scientists and better science. Importantly in the long run it will save time – it’s fine to rush a review, but if you don’t provide detailed feedback you (or someone else) is likely to see the same paper again (or similar papers) which means it will have to be reviewed again. It might even be published second time around when you think it is a poor piece of work. Then someone will have to write a rebuttal or comment to counter the problems that were not fixed because the author did not get good feedback (and these take far longer to produce then a good review, and then they have to be reviewed!). In short, if you do it right the first time and it might take longer, but on average you will save a ton of time. It may not be *your* time saved, but on average everyone should do better out of the deal.

In a different vein, but related to time, if reviews are carried out faster then research turnover will be faster. I can’t refer to a paper that has not been published yet, and if it is sat in review for two years this does no one any favours. The author’s work is held up, which can be a big problem when you are looking for a job or funding, and if the methodology is new it can stop your research dead – you don’t want to push an idea if it might be deeply flawed but you can’t get a response from the community. Your work might be scooped if it sits around waiting for publication and you can miss out on deserved credit, or your work might become out of date and unpublishable simply because it appears after something else even though  it was completed years ago. These are real problems even if they only happen rarely.

The fundamental issue seems to be that of time. I am not trying to suggest that editors and referees fundamentally do a bad job, just that things could be much better. I am well aware that people are constantly facing deadlines and responsibilities that suck up their time, but research papers are the most fundamental aspect of science and they are not being treated as they should. Papers have to be read properly by editors. The right referees have to be selected. They have to referee the papers thoroughly and correctly. The editors must then make a careful decision based on the reviews and the manuscript in hand and pass this onto the author. To do otherwise is, in my opinion, disrespectful to the work of the authors and fundamentally a problem for science. I have certainly learned how hard it can be to find time to do everything I want to or am supposed to in the last two years, but I am still able to prioritise reviews.

If people can just take a bit more time over the editorials and refereeing responsibilities not only will the science be improved, but the time saved in the long run will more than repay it. A journal with a good reputation for review will get more and better papers and improve their standing, something the authors, editors and journals should all desire – so the effects and motivation should be obvious and palpable. I know of plenty of people who actively avoid some journals simply because of the review time. What is required though, is for authors to take journals to task over poor reviews and long waiting times. This again I feel feeds back into the problem of age – young researchers are less likely to do this to avoid conflict and senior researchers are less troubled given their secure positions and having time on their side in terms of resubmissions and long waiting times. Between these two, neither demographic will challenge the journals and the status quo is maintained. Without a concerted effort I can’t see it changing – I don’t have the luxury of being able to boycott journals who I think give bad service (and of course when I am not the lead author it is not my responsibility either, even if I am still affected) or retract papers over slow reviews – I have to try and get things published to improve my CV and get a grant or a job (which in itself is another interlinked problem). In at least some cases this might well be an issue of ‘conflict of interest’ – the referees and editors and authors may want to see a good paper published, but the journals jsut want to see *something* published, and that is a very different issue, and one that naturally and understandably (if unhelpfully) they will push forwards.

Well, that was five times longer than expected or intended, any thoughts and comments? Once more I will stress that I am NOT advocating the end or even a serious overhaul of peer review – the concept is fine, merely its application is at fault. Again, there are plenty of excellent referees out there and excellent editors and journals who turn things around fast and accurately with good, through and well argued reviews and prompt balanced decisions, but there is at least a significant minority who are not. When each paper gets handled by one editor and two or three referees, it only takes one to drag the process out for six months no matter how fast the others are, or to do a rushed and incomplete job for things to get complicated and mistakes made or delays had, so even one person in ten can still affect more than a third of the papers being submitted.