Problems with peer review

There have been a number of posts both here and elsewhere on the mechanism of peer review (how papers are refereed before publication) and various comments coming through on the subject. Based on these and my own discussions with various colleagues as well as my experiences as a referee, editor and author, it seems to me that there are significant problems with the system. (I could have just been unlucky, or it might be really bad for dinosaur papers or palaeo ones, but discussions with colleagues in other disciplines suggest otherwise). However, I would argue that some of them could be alleviated or even fixed with a few relatively simple measures and others could be improved dramatically. Ultimately the real problem I suspect is simply a lack of time and or / funds to do things better and this quite possibly will never change. However, if sufficient people make sufficient noise about a problem, change can happen. This post therefore is going to consist of me putting forward what I think are serious problems with peer review, and ways that these can be improved. Everything is hypothetical (I can hardly be sure they will work, even if they are ever tried) but I think reasonable. I’d be very interested to hear what other professionals think of peer review as it stands and what could or should be done to improve the situation. Obviously in principle, the idea is fine, and many papers are reviewed and refereed properly and correctly, but this is not always the case (indeed could be argued is rarely the case) and this *is* a problem.

Before you start, if you do not know much about peer review and / or have never tried to get a scientific paper published I suggest you read my guides to review and publication (and some of the comments too) as a starting point as to the process that this is about. Right onto the list, which is in no particular order:

1. Proper separation of responsibilities of the editors and referees. It seems that referees are often doing the editors job of checking things like formatting and there is general toe treading which causes confusion and duplication of work. This is related to:

2. Insufficient guidelines for referees. Referees are often just given a paper and told to review it. Even when there is some kind of form to fill in, I have yet to find one especially helpful as I am often faced with yes/no choices for answers when I want to write a sentence or two to explain my decision or add a qualifier. An example was a review I did for a dedicated volume as part of a wider series. The question asked if I felt the paper would be of general interest – to which of course the answer was ‘no’, it was intended for a specialist volume, but then an unthinking (or unaware) editor might see this as a serious negative point against the manuscript. Referees need clear guidelines as to what they should and should not comment on, and be given the freedom to express those opinions even within a framework. Given just how many papers get refereed for the average journal each year, this should not be too much to ask.

3. Poor referee selection. Authors are often asked to submit a list of potential referees for their papers yet in my experience these are rarely selected. This means that I am wasting my time writing out lists of researchers with contact details, and appropriated referees are not being selected. Of course not everyone can referee every paper and editors are limited in their choices, but often entirely unsuitable referees are chosen to review papers. If your paper is based on a cladistic analysis, it should not be refereed by someone who does not use cladistics. If you paper is on dinosaurs, it should not be reviewed by a mammal specialist. It is incredibly frustrating to have a paper rejected when it seems clear that the referee simply did not understand or appreciate what was being done. If you write a paper on dinosaur behaviour, yes you probably want a dinosaur expert as a referee to check the palaeo side of things, but you also need a behavioural specialist to check the ethology. In practice, a palaeo editor at a palaeo journal will pick two palaeontologists since they don’t know of any appropriate referees. It might be beyond reason, but why can’t we have a database of researchers and what things we can (and cannot referee) for editors to check. Referees must also take responsibility here and refuse to review papers for which they are un- or underqualified to review, and demand that appropriate referees are chosen. If not, good papers are rejected because they are not reviewed properly, and bad ones are published for the same reason.

4. Incredibly poor time keeping. I recently had a paper that took eleven months get a response. What was more galling was that the editor apparently had it for six months before he even sent it out for review. Referees often take six months to a year to review a paper. This is unacceptable. Yes referees and editors are busy, and they don’t always have the time to deal with something instantly, it takes time for an editor to find referees willing to do the work and time to sit and write a review. However this is stupid. If a review takes four hours to write it will take four hours to do today or four hours to do two years from now. Editors cannot be allowed to take months to send papers out for review and referees cannot be allowed to take months to write reviews. This is hard to fix since the motivation must come from the journals and editors / referees themselves and I am not sure what this motivation could take. My guess is that a journal that got a reputation for rapid and good refereeing, more people would send their manuscripts there which would improve the quality of the work going into the journal, improve the journal’s quality and popularity and benefit everyone (Nature manages to get review back in two weeks, why can it take more than a year for others?). At the least, the authors need a right to reply to editors or senior editors about review times. This leads to:

5. Improved dialogue with authors. Authors (especially young and inexperienced ones) are unwilling to challenge reviews or question editors over decisions. This is understandable (I would not want to complain about incompetent or unfair reviews to an editor who might review my next paper, or be looking for a student or reviewing my next grant) but also unfortunate. At the least this must be a fundamental part of submitting a paper – the author must be told how to contact the chief editor, or handling editor and offered advice on how to raise questions / file complaints. I am sure editors would hate it, but problems would be far better resolved with far fewer hurt feelings if there was a clearer way to resolve conflicts over publications. A more general dialogue between authors and editors would also help cut down on delays and hopefully force editors to get things done to a schedule and deal with referees better.

6. Tendency to focus on the negative. This might sound silly, but often reviewing forms when sent out are framed such that referees are expected or asked only to note problems with the paper in hand. This can place an undue emphasis on the negative (“what problems must be changed?” vs “what are the strengths and weakness of the paper?”). I have had feedback from referees of papers that were rejected who though the actual work was good and publishable, but since they only send in a list of problems / corrections, the editors did not like it. (This is of course not the only reason, but you take the point I hope). Obviously the reviews will always look at what is wrong, since you don’t need to change or comment on what is good or right, but it would be good if referees in general, and editors / review forms specifically encouraged, people to provide a short list of major strengths or good parts of the paper. This will help the editors make a more balanced decision, and help the authors learn what is considered to be good science, not just what is bad science.

7. Moderating editors’ responses. This harks back to the communication issues and the ability to challenge or at least, request explanations for, decisions. Obviously editors have restrictions and limitations on what can go into a journal – even excellent papers may be rejected if there is simply no room, or the focus of the paper is not of interest to the audience they are targeting. However, I have certainly had judgments from editors that strongly suggest that they have read neither the paper in detail, nor the comments of the referees. If the referees’ opinions strongly conflict the editor must make a balanced judgment, or better yet go to a third referee to get a further opinion. It is frustrating to get a terrible review where you feel the referee has completely missed the point of the paper, claims you have done things you have not, or not done things you have and then find the editor things this is entirely reasonable. If the editor is unsure if a paper is suitable for publication, I am not sure what the harm would be to speak to the author and ask their opinion. They can discuss the issues with the paper and if it can be redirected a little to get it published, I doubt many would be unhappy with this, instead of just getting a flat rejection. The editors can also speak again to the referees, I recently recommended a paper for rejection and later saw it published with barely a change made to it – the editor is of course happy to disagree with my verdict, but I felt and stated that there were a number of basic and indeed critical errors to the paper. I can’t help but think that a better dialogue would have helped get these points across and a poor paper could have been made (I think) into a decent paper.

8. Uninformative reviews. There is no point in rejecting a paper as a referee if you can offer no more than ‘reject this paper’. Editors should *reject* reviews that are short or uninformative. First of all they imply that the paper was not read in detail (which if the case is both a disservice to the author and if the paper is rejected as a result, also unfair) and secondly it does nothing to improve the paper. I have recommended outright rejection for papers that were wrong from the title to the appendices but still offered a few pages of what I hoped were constructive criticism as to how the authors might approach a second approach to the problem even if I thought nothing was salvageable of the manuscript at all. Yes it takes time, but it means that next time the author might write something better, they should have learned something and on average the amount of time wasted will go down if fewer bad papers are submitted and the quality of the science will go up.

All of these issues I can relate to personal experiences with papers I have worked on, or heard from close colleagues and collaborators. Discussions with others suggest that these are far from unique and are frustrations experienced by all ranks in the profession, and not just young researchers (though I think we tend to be the worst affected as noted above), or limited to palaeontology, or even dinosaur and pterosaur research. These are important problems since science (and academia) as a whole depends on the peer review system and if good papers are being rejected and poor ones published this is bad. If papers are being held up unnecessarily for months or years though slow refereeing or incorrect rejections forcing resubmissions this is bad. If people are not learning to improve their work though not getting suitable feedback this is bad. And if referees are reviewing papers they are not qualified to process or are wasting their time by duplicating the work of other this is bad. None of these problems are insurmountable, or even complex, they just require some thought and application.

If reviews are carried out better, then better research will be published. If referees provide more detailed descriptions and better feedback, then researchers will learn to produce better work more quickly. This will produce both better scientists and better science. Importantly in the long run it will save time – it’s fine to rush a review, but if you don’t provide detailed feedback you (or someone else) is likely to see the same paper again (or similar papers) which means it will have to be reviewed again. It might even be published second time around when you think it is a poor piece of work. Then someone will have to write a rebuttal or comment to counter the problems that were not fixed because the author did not get good feedback (and these take far longer to produce then a good review, and then they have to be reviewed!). In short, if you do it right the first time and it might take longer, but on average you will save a ton of time. It may not be *your* time saved, but on average everyone should do better out of the deal.

In a different vein, but related to time, if reviews are carried out faster then research turnover will be faster. I can’t refer to a paper that has not been published yet, and if it is sat in review for two years this does no one any favours. The author’s work is held up, which can be a big problem when you are looking for a job or funding, and if the methodology is new it can stop your research dead – you don’t want to push an idea if it might be deeply flawed but you can’t get a response from the community. Your work might be scooped if it sits around waiting for publication and you can miss out on deserved credit, or your work might become out of date and unpublishable simply because it appears after something else even though  it was completed years ago. These are real problems even if they only happen rarely.

The fundamental issue seems to be that of time. I am not trying to suggest that editors and referees fundamentally do a bad job, just that things could be much better. I am well aware that people are constantly facing deadlines and responsibilities that suck up their time, but research papers are the most fundamental aspect of science and they are not being treated as they should. Papers have to be read properly by editors. The right referees have to be selected. They have to referee the papers thoroughly and correctly. The editors must then make a careful decision based on the reviews and the manuscript in hand and pass this onto the author. To do otherwise is, in my opinion, disrespectful to the work of the authors and fundamentally a problem for science. I have certainly learned how hard it can be to find time to do everything I want to or am supposed to in the last two years, but I am still able to prioritise reviews.

If people can just take a bit more time over the editorials and refereeing responsibilities not only will the science be improved, but the time saved in the long run will more than repay it. A journal with a good reputation for review will get more and better papers and improve their standing, something the authors, editors and journals should all desire – so the effects and motivation should be obvious and palpable. I know of plenty of people who actively avoid some journals simply because of the review time. What is required though, is for authors to take journals to task over poor reviews and long waiting times. This again I feel feeds back into the problem of age – young researchers are less likely to do this to avoid conflict and senior researchers are less troubled given their secure positions and having time on their side in terms of resubmissions and long waiting times. Between these two, neither demographic will challenge the journals and the status quo is maintained. Without a concerted effort I can’t see it changing – I don’t have the luxury of being able to boycott journals who I think give bad service (and of course when I am not the lead author it is not my responsibility either, even if I am still affected) or retract papers over slow reviews – I have to try and get things published to improve my CV and get a grant or a job (which in itself is another interlinked problem). In at least some cases this might well be an issue of ‘conflict of interest’ – the referees and editors and authors may want to see a good paper published, but the journals jsut want to see *something* published, and that is a very different issue, and one that naturally and understandably (if unhelpfully) they will push forwards.

Well, that was five times longer than expected or intended, any thoughts and comments? Once more I will stress that I am NOT advocating the end or even a serious overhaul of peer review – the concept is fine, merely its application is at fault. Again, there are plenty of excellent referees out there and excellent editors and journals who turn things around fast and accurately with good, through and well argued reviews and prompt balanced decisions, but there is at least a significant minority who are not. When each paper gets handled by one editor and two or three referees, it only takes one to drag the process out for six months no matter how fast the others are, or to do a rushed and incomplete job for things to get complicated and mistakes made or delays had, so even one person in ten can still affect more than a third of the papers being submitted.

19 Responses to “Problems with peer review”


  1. 2 David Hone 09/03/2009 at 7:14 pm

    Thanks for that Lemming. That ties in well with things I have said on here about reviews, and has Andy the Open Source Paleontologist on his blog. While these are all from the geo-palaeo side of things, I get similar reports from my friends in chemistry, physics and biology implying much of this is a scinece-wide, if not academia-wide set of issues.

  2. 3 Dave Raikow 09/03/2009 at 10:12 pm

    1, 2, 6, 7, 8: Hear, hear.

    3: Tell me about it. Also frustating is trying to suggest referees for work that is multi-disciplinary, or heaven forbid, innovative.

    4: Do paleontological referees really take that long? I feel bad if I take a few weeks to turn in a review.

    5: The best editor by far I have ever encountered took my first paper, written as an undergraduate, under her wing, and helped me improve it. Rosemary Mackay, too bad she’s retired.

    I suppose every scientist has reviewing horror stories. But perhaps the ultimate propblem is that reivewing is a highly valuable service provided by researchers for free. If we were lawyers you can bet all that work (and it is work, often taking an entire day or more) would be billable. And I don’t just mean we are working for businesses without compensation, reviewing has little or nothing to do with promotion. So, what incentive is there for researchers to work quickly? Reputation?

  3. 4 David Hone 10/03/2009 at 5:15 am

    That is a good point. Certainly I know people who are good editors and referees, and also I know that if I do not know the author or the methods of a paper well, having certain names on there as referees can help boost my opinion of it (or more often lessen concerns). I guess the ultimate problem is the lack of respect it is given as a part of the job by administration people. It is important and must be done properly and let’s face it, it is a prestige thing to be asked to review for Nature etc. it implies you are on the top end of your field.

  4. 5 Rico Tabor 12/03/2009 at 7:22 pm

    I agree totally with this. slow turnover is extremely damaging, particularly in areas where you know other people could publish research which affects your own.

    as always, some journals are better than others, and as you note, it’s no coincidence that the “best” (Nature, Science, and in my field JACS and Angewandte) happen to be the ones that have fast and effective review and editing processes in place.

    other journals take note! immediacy is (in)directly related to prestige. if you have a reputation for rapid publication of important work, this will mean you are more widely read and respected, and also that you will tend to attract a better quality of work.

  5. 6 David Hone 12/03/2009 at 7:26 pm

    Quite, and of all of these ideas, I think this would be the easiest to fix, and have the biggest impact. We must do soemthing.

    Dave

  6. 7 Dr Vector 17/03/2009 at 4:09 am

    Good thoughts, Dave. Sounds like you have some real horror stories–my sympathy. I’ve been fortunate to have mostly gotten competent editors and fast, fair reviewers. Mostly.

    Unfortunately, I also know of a few cases in which particular people have repeatedly sent in reviews late, and these reviews have been hyper-critical to the point of absurdity. I can only assume that these people are trying to hurt their colleagues as much as possible, first by writing ridiculously negative reviews, and second by sitting on those reviews until well after the editorial deadline to slow down the progress of their colleagues’ manuscripts. In at least one case this tactic backfired, because the “torpedo review” came in so late that the editor had already accepted the paper based on the other reviews, and it was so negative that the author could justly claim that it offered no suggestions for how the paper might be improved.

    I am coming to the view that when people abuse the process that badly, they should get called on it, publicly, in blogs and before the relevant professional societies. In my view this would enhance rather than erode collegiality; what destroys collegiality is letting people get away with this blatantly unethical behavior repeatedly. If the offender to whom I’m referring is reading this, please feel free to take that as a warning.

    I also think there should be more public discussion of how journals are run. We working paleontologists write, review, and consume the content of the journals in our field; when one of those journals becomes poorly run, surely when we ought to be able to bring some leverage to bear to get it back on track. Although in the short term it would probably mean some egos would get bruised, in the long run it would be more constructive than just abandoning the journal until it changes hands, which is where some of us are probably at regarding one journal in particular.

    There is also the grass-roots approach to improving peer review, which is for each of us to always turn our reviews around quickly, to always offer _constructive_ criticism, and to sign our reviews as often as possible. I have found that I am much more considerate, thoughtful, and helpful when I sign my reviews. Hopefully we can each cultivate a reputation for being a good reviewer, good not in the sense of always giving favorable reviews, but in the sense of always being prompt and constructive. Every time an author or editor requests a review from us, that’s one less review they’ll get from the foot-dragging, relentlessly negative Axis of Evil Reviewers.

  7. 8 Andy 17/03/2009 at 9:38 pm

    Great post! I too agree. Many of the problems in review times, etc., would be fixed by a “do unto others” approach.

    Regarding the time issue, I have never been hesitant to contact the editors if a review seems to be taking an abnormally long time. In order to avoid being a complete annoyance, I’ll usually ask around to colleagues who have published in a journal (or check the submitted/accepted dates for recently published papers) and find out the timing for typical reviews. If other folks’ experience is two or three months, or six months, then I’ll wait just a little beyond that time before starting to make inquiries. On more than one occasion, I’ve learned that my manuscript was “forgotten,” or that a reviewer hadn’t yet been assigned, or whatever. If I hadn’t bugged the editor, my manuscript likely would have languished for months more!

  8. 9 David Hone 18/03/2009 at 7:08 am

    Thanks for the coments guys. I think after having written this that the major issue os that of timing, if we could fix that then the others become far less relevant (a poor review or editorial process that means you get rejected is not so bad if it happens in 3 weeks and you can resubmit fast elsewhere). I can’t help but think that looking at other fields (including biology and geology) we are really really slow and we can work on this. All that is required is peer pressure. Might have to think about this properly…

  9. 10 Manabu Sakamoto 18/03/2009 at 8:50 pm

    I’ve recently heard through someone else that a senior researcher (a professor to be exact) mentioned that ‘the culture has changed’ in that nowadays, there are a lot of criticism for the sake of criticism.

    Regardless if this remark is true or not, because it is in our nature as humans to criticise other people, it is probably true that publication is becoming increasingly difficult mainly due to the nitpicking criticisms from reviewers…

    There are way too much new technologies that can shed more light into your research, but if only everyone had access or the funding to use every method conceivable…some suggestions are just not feasible.

  10. 11 David Hone 18/03/2009 at 10:07 pm

    Yes, I’d agree with that too. Being told that your paper is not good enough / not well enough supported until you have done FEA / some hideously complex statistical analysis / seen every speciemn etc. is silly. If you have the idea and can support it then it should be published. It may not make the biggest journals, or get more criticism becuase the support is low, but that doesn’t make it wrong or not worthy of publciation and discussion. We can test it with other methods later, but we can’t just stop our research or getin athe extra time / money . experience to start again on a whim. Getting the idea out there lets us all investigate and test it.

  11. 12 Christopher Taylor 20/03/2009 at 12:53 pm

    Lot of food for thought there. Even if the way journals are run isn’t changing, you’ve just given me a few things to keep in mind the next time I’m asked to write a review myself.

    Just one detail I felt like adding – “Uninformative reviews. There is no point in rejecting a paper as a referee if you can offer no more than ‘reject this paper’.” Does anyone else feel that it works the other way, too? A couple of times I’ve had a review come back with “no changes needed, publish as is”, and whenever that’s happened I’m left with the suspicion that the reviewer never (or barely) actually read the manuscript. Especially when the other reviewer suggests a number of required changes…

  12. 13 David Hone 20/03/2009 at 1:46 pm

    Well that situation ‘publish as is’ is very rare. I have had a couple but even they usually at least give a few comments as to minor changes they would like, things that could be added or areas that need a little work. The one excpetion would be a very short paper since you pretty much can write a perfect one (or near enough) when it’s only two pages long. It does breed that suspician though.

  13. 14 christopher 22/03/2010 at 8:07 am

    Hey Dave

    I was following various links about peer review and came across your own blog site. I can certainly empathize with your frustrations about peer-review, it’s a bigger problem than some may think. It’s a thorny one, how do you filter out the junk claims from legitimate, but then how do you filter out the junk science from the real stuff?

    I hope you dont mind, but at my website is a description of my book Convergence, which covers manya areas of science, including peer review. Although I’m a Neuroscientist by training, I’ve had physicists read the book and find it a page turner (see reviews at website).

    Chris.


  1. 1 A quick note on referees / journals again « Dave Hone’s Archosaur Musings Trackback on 15/05/2009 at 10:29 pm
  2. 2 OK, fine, I’ll talk about that pterosaur / albatross paper « Dave Hone’s Archosaur Musings Trackback on 01/06/2009 at 9:13 am
  3. 3 Tremble ye mighty referees and authors « Dave Hone’s Archosaur Musings Trackback on 10/11/2009 at 9:08 am
  4. 4 Dave Hone’s Archosaur Musings Trackback on 11/02/2010 at 4:05 pm
  5. 5 “Any jackass can trash a manuscript….” « Dave Hone's Archosaur Musings Trackback on 24/05/2011 at 7:44 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




@Dave_Hone on Twitter

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 351 other followers


Follow

Get every new post delivered to your Inbox.

Join 351 other followers

%d bloggers like this: