How to review a paper

Continuing with the important theme of academic research and publishing it seemed high time I deal with the opposite side of writing a paper which is reviewing one. This should serve as a guide for those asked to referee a paper without much or any experience, and also for those simply trying to get a paper written and get it past the referees – it often helps to know what is going on on that side of the publishing ‘wall’ as it will help you deal with editors, referee’s comments and so on.

Typically you will get an e-mail from an editor with the title of a paper (and possibly the names of the authors) asking if you can review a paper for a journal. Before you even say yes, take a few things into consideration.

First, make sure you have the time to do it: there’s no point if you about to head into the field or something similar. I once had a paper that (according to the editor) sat with a referee for nine months before he sent it back and said he was too busy to review it (all seven pages that it was). Had he said anything early on another referee would have been found and I could have been shaved half a year. You also want to make sure you are qualified to pass comment – if the paper is some way outside of your field you may not be the best judge of the paper. Even if it is within your area of expertise, it might be tricky (if you work on flight speeds for example, a paper on flight musculature and it’s effects on thrust production, would be only within your persue), so do let the editor know. He would prefer that you acknowledge this before you start as he might have someone more suitable lined up, or could ensure that the second referee was a muscle expert say. Also if possible you should avoid conflicts of interest with papers by close collaborators, students, your supervisor etc. That isn’t always possible, but do bear it in mind.

So assuming you do say yes, you can expect a manuscript to arrive one way or another for you to review. Typically one writes a letter to the editor with a few general comments (often now with, or as part of a form to fill out with various boxes to tick and so on) and then a letter directly to the authors with your comments. It’s important to note that the authors will not see the letter to the editor so you can be, shall we say, more forthright in your letter to the editor (though many referees don’t make this distinction and you can happily expect both barrels in the main review). Make sure you are following any basic instructions about how to compose your replies (e.g. do you return just an electronic letter, or a hardcopy, can you write on a printout of the paper, or ‘track changes in Word’ etc.) general you have a fairly free reign, but do check.

I would advise that if possible you should read the whole manuscript through in one go before you start any real editing (though marking up things like spelling errors at this point is fine). This will give you a good feel for the paper as a whole, and of course you will probably spot and major issues at this juncture and thus will be mindful of them as you go over it in detail.

Now you should check the general requirements of the journal. Is there a word limit or restriction on colour photographs, what is the formatting style for headings, references etc. Are there any special requirements of the refereeing report you should take note of (such as appeal to a general audience etc.). Be mindful of these issues – I one had a paper recommended for rejection because I had not included an abstract. Fair enough, but it was a short paper aimed at the brevia section of a journal that thus did not take abstracts, something I has specifically mentioned in the covering letter. The referee was not doing his job and I got rejected (at least in part) for following the journal’s own requirements.

Now you can start going through the paper in detail and marking down the issues. Everything from spelling mistakes, missing references, bad grammar and punctuation, misconceptions or contradictions, ideas left handing etc.: the works. Don’t just check the text, but the references, figures, figure captions, appendices, graphs etc. as well. Think through the whole paper – are the methods appropriate, is the paper well explained, has the basic background reading and research been done and understood, is everything coherent and easy to follow. Do they need more data or extra analyses? Has anything obvious been overlooked (it is amazing how often people home in and try to solve details and miss the bigger picture or more obvious explanation).

Even great papers can be marred by bad writing, or even too much information (if twenty concepts are being tested, and then cross compared, it might be better to split the thing into several short papers than a single rambling one) too many ideas etc. so just because the ideas are great and well thought out and analysed does not make it a great paper.

Be firm, but be fair. Your job is to try and make the paper better, so if you know of a paper that the authors have missed that supports their ideas, let them know. If there is a serious flaw, or it clashes with an existing theory (that again they have probably missed or they would have said so) then tell them. Don’t reject it for this, but point it out so that they can deal with it themselves. That said, don’t use their paper as a forum for your ideas. Sure, they might have said something that you have already said in a paper, so direct them to it, but I have seen reviews (ironically often sent anonymously) where the referee largely just sent a list of papers that the authors were told to read and integrate with their work, three quarters of them were by the same author and that kinda implies that the referee just wanted to bump up his citations in my colleague’s paper. Don’t do this.

List all the major and minor errors in your reports, but do tailor it according to the quality of the paper. If the thing is a complete write off don’t feel the need to catalogue every minor error – if the results are completely flawed, or the wrong analytical method was applied to the data set, it doesn’t really matter that they spelt ‘incongruous’ wrong on page 18: just write the report based on the substantive errors since the paper will be rejected or have to go through a second round of review and these minor points can be left for later.

Typically as part of the report one is expected to overall place the paper in one of four categories based on the *overall* assessment of the paper (though some journals have differing standards, or the editors may make the decision themselves without your recommendations). Those who have already read ‘how to write a paper’ will now these already, but they are: publish, minor corrections, major corrections (i.e. resubmission) and rejection.

The first is pretty rare but does happen, and is given where there are only a few minor errors (a reference missing, a few spelling errors etc.) the paper can pretty much go straight though. Minor revisions mean that there are some problems (such as an analysis needs to be redone or rechecked, or there are large numbers of very minor issues) but once the authors have fixed these they need only be checked by the editor and then the paper can join the queue for publication. Major errors (such as inappropriate analyses, substantive revisions to the methods, rewritten discussion etc.) require that large sections of the manuscript be significantly revised and the paper will be sent back to the referees again to recheck things in detail. Finally, if the manuscript is just well, bad, then it will be rejected. (Note that obviously high ranking journals like Nature reject tons of stuff no matter how good because they simply don’t have room for it all, or time for each paper to get a full review – a flat rejection does not always mean the paper was just terrible, though I have been sent enough terrible papers to review).

At this point I like to leave the thing a few days and then go back and read the paper again, and then my report and comments. Often you will pick up a few extra bits and bobs which is of course good, but it can also give you a new appreciation for the paper. Often I have found that I have a new appreciation for some points in the paper and have often had to go back over my reports and delete or amend sections as I realise the mistake in interpretation was mine, not that of the authors.

Now you can give the report a final check and make your recommendation to the editor. Give an honest and genuine opinion as to the worth of the piece and its suitability (or otherwise) for publication in the journal. Don’t be afraid to make strong recommendations, or weak ones – if you are genuinely not sure of the quality of the paper, or think that you are underqualified to judge important parts of it then do tell the editor and suggest he refer the paper to another referee or check things himself. Your job is to help review and improve the paper and to get good science done, not to be the judge per se of the merits of the paper, so if you don’t know, say so.

One last point to be made is that one generally has the choice to be anonymous when submitting the reports (to the authors that is, obviously the editor knows who you are). Typically most referees reject this option – it’s only fair the other person who is writing what about them, but others prefer to be anonymous all the time, and of course there are always situations where anonymity is prudent (such as giving a review for someone you are applying to work for, for example).

That should just about do it. This is an important aspect of being a scientist and it should be taken seriously – authors make mistakes and this is the main opportunity for them to be caught before publication and you are doing the author, editor and science a disservice if you skimp. Sure authors are responsible for their papers, but the point of peer review is for there to *be* a review, so make sure you do your job as best you can. There are few things more depressing that getting a review back after six months which is five liens long and just says that your paper is bad. OK, fine, you think it’s junk, but please have the courtesy to spend and extra 30 minutes to write down *why* you think its wrong and *how* it can be improved, or where I can start again to do something better next time. That doesn’t help me learn anything, and is thus a waste of time.

This is in itself quite a document and hopefully will provide a decent guide for others to completing a review. It really is quite straight forward and simple to do, but this should offer a few hints and tips. As you have probably noticed, this is quite a prolonged process. A complex paper of a few dozen pages can take a couple of days to review properly and if you have to start checking appendices, or rerunning an analysis to make sure things are accurate, or maybe even hunt down and read a couple of papers to cross-check important points it can of course take longer still. Please take the time and do it right: all of science will benefit from a good paper being made better before being published.

As ever with these ‘how to’s, comments are welcome from fellow professionals if there are things I have obviously missed or you strongly disagree. I should also add that between me writing this and posting it, the Open Source Paleontologist has had the same idea and you can read his series of posts on this very topic. LINKS What I especially like is while we seem to have covered all of the same salient points, but in rather different ways and with different emphasis so the two are rather nice companion pieces and I would highly recommend you read both.