How to review a paper

Continuing with the important theme of academic research and publishing it seemed high time I deal with the opposite side of writing a paper which is reviewing one. This should serve as a guide for those asked to referee a paper without much or any experience, and also for those simply trying to get a paper written and get it past the referees – it often helps to know what is going on on that side of the publishing ‘wall’ as it will help you deal with editors, referee’s comments and so on.

Typically you will get an e-mail from an editor with the title of a paper (and possibly the names of the authors) asking if you can review a paper for a journal. Before you even say yes, take a few things into consideration.

First, make sure you have the time to do it: there’s no point if you about to head into the field or something similar. I once had a paper that (according to the editor) sat with a referee for nine months before he sent it back and said he was too busy to review it (all seven pages that it was). Had he said anything early on another referee would have been found and I could have been shaved half a year. You also want to make sure you are qualified to pass comment – if the paper is some way outside of your field you may not be the best judge of the paper. Even if it is within your area of expertise, it might be tricky (if you work on flight speeds for example, a paper on flight musculature and it’s effects on thrust production, would be only within your persue), so do let the editor know. He would prefer that you acknowledge this before you start as he might have someone more suitable lined up, or could ensure that the second referee was a muscle expert say. Also if possible you should avoid conflicts of interest with papers by close collaborators, students, your supervisor etc. That isn’t always possible, but do bear it in mind.

So assuming you do say yes, you can expect a manuscript to arrive one way or another for you to review. Typically one writes a letter to the editor with a few general comments (often now with, or as part of a form to fill out with various boxes to tick and so on) and then a letter directly to the authors with your comments. It’s important to note that the authors will not see the letter to the editor so you can be, shall we say, more forthright in your letter to the editor (though many referees don’t make this distinction and you can happily expect both barrels in the main review). Make sure you are following any basic instructions about how to compose your replies (e.g. do you return just an electronic letter, or a hardcopy, can you write on a printout of the paper, or ‘track changes in Word’ etc.) general you have a fairly free reign, but do check.

I would advise that if possible you should read the whole manuscript through in one go before you start any real editing (though marking up things like spelling errors at this point is fine). This will give you a good feel for the paper as a whole, and of course you will probably spot and major issues at this juncture and thus will be mindful of them as you go over it in detail.

Now you should check the general requirements of the journal. Is there a word limit or restriction on colour photographs, what is the formatting style for headings, references etc. Are there any special requirements of the refereeing report you should take note of (such as appeal to a general audience etc.). Be mindful of these issues – I one had a paper recommended for rejection because I had not included an abstract. Fair enough, but it was a short paper aimed at the brevia section of a journal that thus did not take abstracts, something I has specifically mentioned in the covering letter. The referee was not doing his job and I got rejected (at least in part) for following the journal’s own requirements.

Now you can start going through the paper in detail and marking down the issues. Everything from spelling mistakes, missing references, bad grammar and punctuation, misconceptions or contradictions, ideas left handing etc.: the works. Don’t just check the text, but the references, figures, figure captions, appendices, graphs etc. as well. Think through the whole paper – are the methods appropriate, is the paper well explained, has the basic background reading and research been done and understood, is everything coherent and easy to follow. Do they need more data or extra analyses? Has anything obvious been overlooked (it is amazing how often people home in and try to solve details and miss the bigger picture or more obvious explanation).

Even great papers can be marred by bad writing, or even too much information (if twenty concepts are being tested, and then cross compared, it might be better to split the thing into several short papers than a single rambling one) too many ideas etc. so just because the ideas are great and well thought out and analysed does not make it a great paper.

Be firm, but be fair. Your job is to try and make the paper better, so if you know of a paper that the authors have missed that supports their ideas, let them know. If there is a serious flaw, or it clashes with an existing theory (that again they have probably missed or they would have said so) then tell them. Don’t reject it for this, but point it out so that they can deal with it themselves. That said, don’t use their paper as a forum for your ideas. Sure, they might have said something that you have already said in a paper, so direct them to it, but I have seen reviews (ironically often sent anonymously) where the referee largely just sent a list of papers that the authors were told to read and integrate with their work, three quarters of them were by the same author and that kinda implies that the referee just wanted to bump up his citations in my colleague’s paper. Don’t do this.

List all the major and minor errors in your reports, but do tailor it according to the quality of the paper. If the thing is a complete write off don’t feel the need to catalogue every minor error – if the results are completely flawed, or the wrong analytical method was applied to the data set, it doesn’t really matter that they spelt ‘incongruous’ wrong on page 18: just write the report based on the substantive errors since the paper will be rejected or have to go through a second round of review and these minor points can be left for later.

Typically as part of the report one is expected to overall place the paper in one of four categories based on the *overall* assessment of the paper (though some journals have differing standards, or the editors may make the decision themselves without your recommendations). Those who have already read ‘how to write a paper’ will now these already, but they are: publish, minor corrections, major corrections (i.e. resubmission) and rejection.

The first is pretty rare but does happen, and is given where there are only a few minor errors (a reference missing, a few spelling errors etc.) the paper can pretty much go straight though. Minor revisions mean that there are some problems (such as an analysis needs to be redone or rechecked, or there are large numbers of very minor issues) but once the authors have fixed these they need only be checked by the editor and then the paper can join the queue for publication. Major errors (such as inappropriate analyses, substantive revisions to the methods, rewritten discussion etc.) require that large sections of the manuscript be significantly revised and the paper will be sent back to the referees again to recheck things in detail. Finally, if the manuscript is just well, bad, then it will be rejected. (Note that obviously high ranking journals like Nature reject tons of stuff no matter how good because they simply don’t have room for it all, or time for each paper to get a full review – a flat rejection does not always mean the paper was just terrible, though I have been sent enough terrible papers to review).

At this point I like to leave the thing a few days and then go back and read the paper again, and then my report and comments. Often you will pick up a few extra bits and bobs which is of course good, but it can also give you a new appreciation for the paper. Often I have found that I have a new appreciation for some points in the paper and have often had to go back over my reports and delete or amend sections as I realise the mistake in interpretation was mine, not that of the authors.

Now you can give the report a final check and make your recommendation to the editor. Give an honest and genuine opinion as to the worth of the piece and its suitability (or otherwise) for publication in the journal. Don’t be afraid to make strong recommendations, or weak ones – if you are genuinely not sure of the quality of the paper, or think that you are underqualified to judge important parts of it then do tell the editor and suggest he refer the paper to another referee or check things himself. Your job is to help review and improve the paper and to get good science done, not to be the judge per se of the merits of the paper, so if you don’t know, say so.

One last point to be made is that one generally has the choice to be anonymous when submitting the reports (to the authors that is, obviously the editor knows who you are). Typically most referees reject this option – it’s only fair the other person who is writing what about them, but others prefer to be anonymous all the time, and of course there are always situations where anonymity is prudent (such as giving a review for someone you are applying to work for, for example).

That should just about do it. This is an important aspect of being a scientist and it should be taken seriously – authors make mistakes and this is the main opportunity for them to be caught before publication and you are doing the author, editor and science a disservice if you skimp. Sure authors are responsible for their papers, but the point of peer review is for there to *be* a review, so make sure you do your job as best you can. There are few things more depressing that getting a review back after six months which is five liens long and just says that your paper is bad. OK, fine, you think it’s junk, but please have the courtesy to spend and extra 30 minutes to write down *why* you think its wrong and *how* it can be improved, or where I can start again to do something better next time. That doesn’t help me learn anything, and is thus a waste of time.

This is in itself quite a document and hopefully will provide a decent guide for others to completing a review. It really is quite straight forward and simple to do, but this should offer a few hints and tips. As you have probably noticed, this is quite a prolonged process. A complex paper of a few dozen pages can take a couple of days to review properly and if you have to start checking appendices, or rerunning an analysis to make sure things are accurate, or maybe even hunt down and read a couple of papers to cross-check important points it can of course take longer still. Please take the time and do it right: all of science will benefit from a good paper being made better before being published.

As ever with these ‘how to’s, comments are welcome from fellow professionals if there are things I have obviously missed or you strongly disagree. I should also add that between me writing this and posting it, the Open Source Paleontologist has had the same idea and you can read his series of posts on this very topic. LINKS What I especially like is while we seem to have covered all of the same salient points, but in rather different ways and with different emphasis so the two are rather nice companion pieces and I would highly recommend you read both.

16 Responses to “How to review a paper”


  1. 1 Bill Parker 04/01/2009 at 10:38 am

    >>Now you should check the general requirements of the journal. Is there a word limit or restriction on colour photographs, what is the formatting style for headings, references etc. Are there any special requirements of the refereeing report you should take note of (such as appeal to a general audience etc.). Be mindful of these issues – I one had a paper recommended for rejection because I had not included an abstract. Fair enough, but it was a short paper aimed at the brevia section of a journal that thus did not take abstracts, something I has specifically mentioned in the covering letter. The referee was not doing his job and I got rejected (at least in part) for following the journal’s own requirements.<<

    I really don’t feel that format checking is the job of the reviewer, instead this is the job of the editor and something that should be done before the paper is even sent out for review. This approach would have saved you having your paper rejected for not having an abstract when none was required. I think it is ridiculous that your paper was rejected for this reason and hopefully you filed an immediate appeal.

    When roles are not clearly defined I think that there is often an unintentional blur between the role of the editor and the role of the reviewer, which can often cause delays and problems. The reviewer’s job is the make sure that the paper is technically sound, well-organized, and has included all relevant literature. It is the editor’s job to make sure the paper conforms to the journal’s style guideline,includes all cited references, and has correct spellings and grammar.

  2. 2 David Hone 04/01/2009 at 12:25 pm

    I would agree that this should primarily be the job of the editor, but I don’t see why the referee can’t take an active interest in this. I guess the problem is as you say primarily one of definition of roles. I’ve been the referee for probably a couple of dozen papers and msot come without any form of instruction at all which is pretty shocking given how prestigious some journals are and how long they have been running. In my experience few editors even read the papers in detail (if at all on occasion) let alone providing feedback on spelling and formatting. Some do certainly, but many do not. Let’s face it peer review could be a lot better, but until we compling long and loud about it, it’s not goign to cahnge (sadly). I think I’ll have to start an open thread on this at soem point as it seems to me that a great many people think that eer review is very badly carried out, but, and this is very important, it could be greatly improved with a few simply defined and implemented changes.

  3. 3 John Hutchinson 04/01/2009 at 5:19 pm

    Hi, as an editor and as a fairly frequent reviewer I agree with the main points. I find stories of reviewers/editors delaying papers for months to be quite disturbing. I believe one job of reviewers and editors is to speed the pace of publishing.

    Especially considering the modern capacity for rapid online publishing, there is little excuse for long delays. And in the competitive current economy, journals that move slowly will be eclipsed sooner or later. For example I see little reason to bother with journals that do hard-copy-only reviews, they are a thing of the past. Research has a shorter shelf-life these days before the next discovery changes the story; science is moving fast.

    I am lucky(?) to only have to handle 2-3 papers/month as an editor so I can understand how some editors can get more buried than me; hence some cases are understandable, in moderation.

    But I have little sympathy for slow reviewers. Once you agree to a review you’ve made a social contract to finish it efficiently. And in my experience most reviews can be done in 1-2 weeks (taking maybe 2-3 hrs at most, total) no matter how busy you are. I tend to do them within 24-72 hrs, much as I tend to finish my editorial duties for any paper within that window.

    If you somehow are too busy for a 1-2wk turnaround, you’re either de-prioritizing your responsibility as reviewer (e.g. to get your own work done, thus holding up others’ work), or managing your time poorly, or accepting review invitations too casually. In all cases it’s irresponsible, and I have a hard time understanding it. I have a young daughter, busy job, social life, travel, etc. too.

    Good journals/editors keep “naughty” and “nice” lists of reviewers. Some reviewers get blacklisted because they have discharged their duties poorly. Whereas others become “go-to-guys/gals” that are known for efficient review. The latter can become quite influential in the field- good peer review can be a powerful force for canalizing research progress. But of course “good” is in the eye of the author/editor/reviewer, and bad peer review is worse than none at all.

    Along those lines, it seems that there has been a proliferation lately of poorly, or not at all, peer-reviewed (or quasi-reviewed) books/journals -not to name any names- in paleontology. Perhaps too much is being published. I’m a firm believer in the principle of “publish all research you do or else your research is worthless” (unpublished research is intellectual masturbation). I also believe in citing as much as I can, even if it is in the non-reviewed “grey literature” or just plain dumb. Some might even say I’ve made a career out of demolishing such straw men? :)

    But at some point the community may need to take a more specific stand on which publications to acknowledge or not. I know some researchers that refuse to cite (or even read) papers that are poorly/unreviewed, and perhaps there is some merit to that.

    Anyway interesting thread, end rant.

  4. 4 David Hone 04/01/2009 at 9:24 pm

    Wow, thanks John, good to see it’s not just us at the bottom feeling like this. As it happens I have spent the afternoon commiting my thoughts to paper (well, Word) and I will go through and put this up online for discussion once I have dealt with the mass of pterosaur stuff that is coming through next week. Stay tuned for more.

  5. 5 Andy 04/01/2009 at 11:25 pm

    >But at some point the community may need to take a more specific stand on which publications to acknowledge or not. I know some researchers that refuse to cite (or even read) papers that are poorly/unreviewed, and perhaps there is some merit to that.

    I’m not sure how I feel about this. . .on the one hand, there are those “publications” (“Tyrannosaurus stanwinstonorum,” anyone?) that are so far out that they don’t even really qualify as publications. These *are* safely (and rightfully) ignored. But on the other hand, I don’t know if it’s intellectually honest to just ignore a paper that may be relevant to the ideas presented in one’s own paper just because the citation is from a crappy journal. From this point, it’s just a little too easy to slip into “I won’t cite this article because the author is a jerk” or “I won’t cite this article because the author disagrees.” Are we any better than the sloppy editors of the third rate journals in this case? And, sometimes useful and relevant papers do get published in funky journals.

    In the end, I tend to side with John on the “cite everything relevant” approach. . .citations are cheap, after all. Maybe I’ll change my mind sometime, but the Aetosaur/Aerosteon sagas of the past year have made me wary. Or at least wary of working on any taxa that begin with “Ae.” No more Aegtyptosaurus for me. . .

  6. 6 David Hone 05/01/2009 at 3:04 pm

    Citations are cheap, though of course this leads to an almost citations arms race where people seek to scite everything on a subject ever. As all kinds of obscure, old and foreign journals become more accessible this can get out of hand. When much of that work is of dubious quality it gets worse. Yes, some paper may be published citing a point contradictory to yours, but if the paper is deeply flawed and out of date, do you include it? I would sya yes, but then you have to counter the point that was made, or the quality of the research and to do that can take a few pages (or even a whole paper in itself). I’m not necessarily advocating jsut ignoring the bad stuff in general, but in soem circumtances it’s more practical to ignore it than to write a mutlu page rebuttal or cite it and have to deal with the consequences.

  7. 7 Rico Tabor 05/01/2009 at 9:04 pm

    another very good article Dave. we handle the administrative work for our boss, who is one of the editors of JCIS, handling around 30 papers per month i guess. i would say we get more badly-written or useless reviews than bad papers, so i hope people take note of this column!

    it’s interesting to note the differences between your field and mine – in chemistry there is usually a fairly well-enforced time limit to review (we give people 21 days to complete once they’ve agreed, otherwise they are politely uninvited) and all reviews are always anonymous.

    we do however share the problem of strange and obscure semi-reviewed “allow anything” journals appearing, which a lot of new science is going into, rather worryingly.

  8. 8 Andy 05/01/2009 at 10:42 pm

    I guess it would depend in part on the topic of the paper–I do agree that it is possible (as in all things) to carry it to a ridiculous extreme. It’s probably much easier to cite and discuss comprehensively for a paper on functional morphology of the eyeball in Protoceratops than a paper on feeding behavior in Tyrannosaurus, for instance. All things in moderation. . .

  9. 10 Michael P. Taylor 06/01/2009 at 7:02 pm

    Thanks Dave, this was useful. I respect your one-or-two-day cooling off period and re-read of the paper — I admit that I don’t do that when reviewing. Although I take issue with this bit: “[I] have often had to go back over my reports and delete or amend sections as I realise the mistake in interpretation was mine, not that of the authors.” I take the line that it’s the authors’ job to damned well make SURE I get it, and on the first read-through. If I, as a careful reviewer, don’t understand, then neither will casual readers.

  10. 11 Oliver 06/01/2009 at 11:17 pm

    @Michael…

    I find that if I’m reviewing/proofing text then I read it very differently, actively trying to spot problems. This can lead to misunderstandings and prejudices which wouldn’t have arisen had I just read it as a normal article.

    Given that, I think a cooling off period can be a very good idea.

  11. 12 David Hone 07/01/2009 at 11:18 am

    Mike, I do take your point, but I do find it happens often to me, and I think it’s often my fault. I can think to linearly and that is not necessarily the author’s fault. In this case I would usually add a comment to the revuiew saying it should be clearer, but this is far better than me reading it once and leaving a comment saying the author has got it wrong when in truth he has not. THough obviously I didn’t make *that* very clear!

  12. 13 Andy 09/01/2009 at 10:13 pm

    I think I’ll side with Dave on this one too. . .in at least 75 percent of the cases where I don’t understand something, it’s because I had missed something that was well-written in the first place. I’ve noticed that in some cases it might even be an emotional result – sometimes something as selfish as not citing me (the reviewer) when I feel I should have been cited, or the authors “scoop” me on a minor idea that I had thought about but never pursued (so I can’t legitimately complain!). This kind of stuff might lead me to a bit of distraction and misunderstanding on the text – which is why I always let my reviews sit for a little while before sending them in, and why I carefully read any especially critical reviews before sending them in. Gotta give a chance for my rational side to reboot. I don’t want to be “that guy” (and we’ve all had “that guy” [in the general sense of a disgruntled reviewer, not a specific scientist] review our papers before)!


  1. 1 Avoiding abstracts « Dave Hone’s Archosaur Musings Trackback on 03/04/2009 at 8:57 am
  2. 2 Twitter Weekly Updates for 2010-02-15 - Kan Ouivirach Trackback on 15/02/2010 at 1:17 am
  3. 3 Useful Links | Dave Attempts a PhD Trackback on 08/10/2012 at 1:45 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




@Dave_Hone on Twitter

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 339 other followers


Follow

Get every new post delivered to your Inbox.

Join 339 other followers

%d bloggers like this: