How to write a useful manuscript review

A “good” peer review is an analysis that is useful and constructive for both the editor and the authors. It helps the editor decide whether a paper should be published, and which changes they should request or require. It helps the author by offering guidance on how to improve their work so that it is clearer and more compelling for a reader. But keep in mind:

Peer review isn’t just criticism—it’s triage.

“Triage” comes originally from military medicine. When wounded soldiers are brought into a medical unit, busy doctors must separate who is likely to die regardless of what surgeons might do from those who can be saved by appropriate medical care.

All manuscripts come into journal offices as “wounded soldiers.” I’ve authored 175 papers, written hundreds of reviews, and handled about 2,000 manuscripts as an editor. Across all those, not a single paper has ever been accepted outright—not one. Some only needed a light bandage, others required major surgery, but they all needed some editorial care.

When a paper is submitted, the editor and reviewers must therefore do triage: does this paper stand a chance of becoming “healthy” and publishable? Or is it so badly “wounded”—damaged by a poor study design, inadequate data analysis, or a weak story—that it should be allowed to die in peace (i.e. be rejected)? An editor at a top-tier journal such as Nature is like a surgeon on a bloody battlefield, getting a flood of patients that overload any ability to treat them all, and so a higher proportion must be rejected and allowed to die. At a specialist journal, the flood is less, and so we can “treat,” and eventually publish, a greater proportion of the papers.

Typically, an editor makes a first triage cut—if the paper is so badly off that it obviously has no chance of surviving, he or she will usually reject the paper without getting external reviews. At Soil Biology & Biochemistry we call that “desk reject;” at Ecology, it’s “reject following editorial review” (ReFER), to emphasize that the paper was reviewed by at least one highly experienced scientist in the field.

But triage doesn’t end with the editor. When you are asked to review a manuscript, the first question you must address is the triage question: is this paper salvageable? Can it reach a level of “health” that it would be appropriate to publish in the journal following a reasonable investment of time and energy on the part of the editorial and review team? A paper may have a dataset that is fundamentally publishable but an analysis or story in such poor shape that it would be best to decline the paper and invest limited editorial resources elsewhere.

Thus, when you are writing a review, the first paragraph(s) should target the triage decision and frame your argument for whether the paper should be rejected or should move forward in the editorial process. Is the core science sound, interesting, and important enough for this journal? Is the manuscript itself well enough written and argued that with a reasonable level of revision it will likely become publishable? If the answer to either of those questions is “no” then you should recommend that the editor reject the paper. You need to explain your reasoning and analysis clearly and objectively enough that the editors and authors can understand your recommendation.

If you answer “yes” to both central questions—the science is sound and the paper well enough constructed to be worth fixing—you move beyond the diagnosis phase to the treatment stage: everything from there on should be focused on helping the authors make their paper better. That doesn’t mean avoiding criticism, but any criticism should be linked to discussing how to fix the problem.

This section of the review should focus on identifying places where you think the authors are unclear or wrong in their presentations and interpretations, and on offering suggestions on how to solve the problems. The tone should be constructive and fundamentally supportive. You’ve decided to recommend that the “patient” be saved, so now you’re identifying the “wounds” that should be patched. It doesn’t help to keep beating a paper with its limitations and flaws unless you are going to suggest how to fix them!  If the problems are so severe that you can’t see a solution, why haven’t you argued to reject the paper?

In this section, you are free to identify as many issues as you wish—but you need to be specific and concrete. If you say “This paragraph is unclear, rewrite it,” that won’t help an author—if they could tell why you thought the paragraph was unclear, they probably would have written it differently in the beginning! Instead say “This is unclear—do you mean X or do you mean Y?” If you disagree with the logic of an argument, lay out where you see the failing, why you think it fails, and ideally, what you think a stronger argument would look like.

It is easy to fall into the “Curse of Knowledge”: you know what you know, so it’s obvious to you what you are trying to say. But readers don’t know what you know! It may not be obvious to them what you mean—you must explain your thinking and educate them. That is as true for the review’s author as for the paper’s author. It’s easy to get caught up in a cycle where an author is unclear, but then a reviewer is unclear about what is unclear, leaving the author flailing trying to figure out how to fix it! A good review needs to be clear and concrete.

Remember, however, that it is not a reviewer’s job to rewrite the paper—it’s still the authors’ paper. If you don’t like how the authors phrased something, you can suggest changes, but you are trying to help, not replace, the authors. If the disagreement comes down to a matter of preference, rather than of correctness or clarity, it’s the author’s call.

When I do a review, I usually make side notes and comments as I read the paper. Then I collect my specific comments, synthesize my critical points about the intellectual framing of the paper, and write the guts of the review—the overall assessment. I target that discussion toward the editor, since my primary responsibility is to help her with triage. She will ultimately tell the authors what changes they should make for the paper to become publishable. Then, I include my line-by-line specific comments. Those are aimed at the authors, as they tend to be more specific comments about the details of the paper. The specific comments typically run from half a page to a few pages of text.

Sometimes reviews get longer—I have written 6-page reviews, reviews where I wanted to say that I thought the paper was fundamentally interesting and important, but that I disagreed with some important parts of it and that I wanted to argue with the authors about those pieces. I typically sign those reviews because a) I figure it will likely be obvious who wrote it, and b) I am willing to open the discussion with the authors: this isn’t an issue of right-or-wrong, but of opinion and where I think that the science might be best advanced by having the debate.

How to offer a specific recommendation?

Accept: The paper is ready to publish. You should almost never use this on a first review.
Accept following minor revision: The paper needs some polishing, but doesn’t need a “follow-up visit”—i.e. you don’t think it will need re-review.
Reconsider following revision: The paper is wounded, but savable. The problems go beyond clarity or minor edits; the paper requires some rethinking. It will therefore likely need re-review. If you recommend “reconsider,” I hope you will also agree to do that re-review.
Reject: The paper should be allowed to die. Either it is fatally flawed in its scientific core, or the scientific story is so poorly framed or written that it is not worth the editorial team’s investment in working to try to make it publishable.

Keep in mind that as a reviewer, you are typically anonymous. The editor is not. If there really are deep flaws in a paper, give me cover by recommending “reject”! If I choose not to take that advice, it makes me the good guy and helps me push the authors to fix the problems: “Reviewer #1 suggested declining the paper, but I think you might be able to solve the problem, so I’ll give you a chance to try.” That of course implies: “but if you don’t, I will reject it.” If you try to be nice and recommend “reconsider” and I decide instead to reject, then it’s all on me and I’m the bad guy. I signed on to do that job, but I do appreciate your help. Give your most honest and accurate assessment but remember that the editor must make the decision and must attach their name to that decision.

Reviewing Revisions

How does this advice change if you are getting a revised manuscript back for re-review? I’ve seen reviewers get annoyed that authors didn’t do exactly what they had recommended. Don’t. First, remember that the editor likely received two or three external reviews that might have varied in their assessments and recommendations—editors need to synthesize all that input before making a decision and offering guidance to the authors. Then, authors might have different ideas about how to solve the problems and to address reviewers’ concerns. In my experience, reviewers are usually right when they identify problems, but are less reliably so in their suggestions for how to fix them. Authors may often come up with different solutions, and it’s their paper! As long as the authors’ solution works, it works. When doing a re-review, your job is to determine whether the paper has crossed the threshold of acceptability, not whether the authors have done everything that you had suggested, and particularly not whether they did everything in the way you might have suggested. In the triage model, the question is not whether the patient is 100% healed, but are they are healthy enough to release?

The more difficult call is when a paper has improved, but not enough. I expect a paper that starts at “reconsider” to step up to “minor revisions” en route to “accept.” But what if you would rate the paper as needing additional major revisions before it closes on acceptability? The paper might have gotten better, but not enough and the trajectory is looking relatively flat. In such a case, you should probably recommend rejecting the paper. It’s not that the paper can’t become publishable, but having given the authors the advice to improve the paper, they either chose not to take it or couldn’t see how to. Well, too bad for them. You can’t write the paper for them and you can’t force the issue; we all have finite time and energy to invest in a patient that isn’t getting better. At some point, we just have to make the hard call, move them out of the hospital ward, say “I’m sorry,” and let them go.

To wrap up, remember that reviewing is a professional obligation—it’s what we do for each other to advance our science. We help our colleagues by identifying areas where the work is unclear or the arguments weak. Review can be a painful process, but writing science is hard; no one ever gets it completely right on the first shot. No one. Ever*. We all rely on peer review, so embrace the process when you’re being reviewed, and do the best job you can when you are the reviewer.

* At least never in my 30 years of experience.

Advertisement