How to Review a Paper You Were Asked to Review

Reviewing is a substantial part of academic work that almost nobody is trained to do. Most people learn by imitating the reports they have received, which propagates whatever they happened to receive.

Practices vary by field: single- or double-anonymous, whether reports are published, how much weight conference reviewing carries. Check the journal's guidance; what follows is general. For a concrete operational example of how a workplace tool approaches a related coordination problem, see the full explanation.

What the editor needs

The editor is making one decision, and your report should support it.

Is this publishable in this venue, and if not, what would have to change?

They need a recommendation — reject, major revision, minor revision, accept — and they need it to be consistent with your comments. A report full of fundamental objections ending in "minor revision" is unusable.

They need to know how confident you are, and where your expertise ends. "I cannot assess the econometrics" is genuinely useful, because it tells the editor to weight another reviewer.

Most journals ask for confidential comments to the editor. Use them for the recommendation and for anything you would not put to the author — a suspected overlap with published work, a concern about the data, or a note that your expertise is partial.

What the author needs

Something they can act on.

Specific. "The identification is weak" is not usable. "The variation could also reflect X; here is why that threatens the interpretation" is.

Prioritised. Distinguish what would change the conclusion from what would strengthen the paper from what is presentational. Delivering all three in the same register makes the report impossible to prioritise, and authors then either over-correct or ignore it.

About the paper they wrote. Reviewing a paper for not being the paper you would have written is the most common failure. Take the question as given.

Including what works. Not politeness — it tells the author what to protect in revision, and revisions frequently damage the strong parts because nobody said which they were.

A structure that works

One paragraph summarising the paper in your own words. This demonstrates you read it, and if you have misunderstood something, the author learns that the exposition failed.

One paragraph on the contribution — what is genuinely new, and how much it matters.

Major points, numbered. Three to six. Each one developed enough to act on.

Minor points, numbered. Typos, unclear sentences, missing citations. Brief.

Numbering matters because the author will respond point by point and the editor will check it. See responding to reviewers.

How long it should take

More than people think and less than they fear.

A first read to understand. Then time away. Then a second read to interrogate.

Several hours for a serious paper, spread over more than one sitting. A report written in forty minutes reads as one.

If you cannot give it that, decline. Promptly, so the editor can find someone else. A late bad review is worse than an early refusal. See declining a review request.

What to actually check

Varies by field; the general shape:

Is the question clearly stated, and is it worth answering?

Does the design answer it? This is the central question in most reviews and it is where the substantive comments come from.

Are the claims supported by what is shown? Overclaiming is common and it is the most useful thing a reviewer catches — frequently the paper is fine and the abstract says more than the results do.

Are the alternative explanations addressed?

Is it reproducible in principle? Are the data, code and materials described sufficiently that someone could check. See sharing data and materials.

Is the literature engaged with, rather than listed? And is anything important missing — genuinely important, not merely yours.

Are the limitations acknowledged?

Things not to do

Do not require citations to your own work unless it is genuinely necessary. Reviewers do this and authors notice, including in anonymous review.

Do not demand a different paper. "The authors should instead have studied Y" is not a review comment.

Do not ask for endless robustness checks. Each one is individually reasonable and twenty of them are a way of rejecting without saying so. Ask for the ones that would change your assessment.

Do not be rude. The author is a person and, in most fields, will eventually know who reviewed them. More importantly: harshness does not make a review rigorous, and a report can be devastating in substance and civil in tone.

Do not review outside your competence without saying so. See reviewing outside your exact area.

Do not use the paper. Material under review is confidential. This should not need saying and it does.

Do not delegate to a student without telling the editor. Co-reviewing with a student is valuable training and it needs to be disclosed; many journals permit it explicitly.

Recommending

Reject where the contribution is insufficient for the venue or the design cannot answer the question. Say which, clearly — an author can act on the second and not the first.

Major revision where the paper could work but needs substantial change. Be specific about what.

Minor revision where you would accept it after presentational changes.

Accept as is is rare and it should exist. If you would publish it unchanged, say so.

Be honest rather than generous. A recommendation softer than your comments makes the editor's job harder and prolongs the process for the author. A rejection that arrives quickly, with reasons, is a service.

The uncomfortable part

Reviewing is unpaid, invisible, and it is what makes the system work. Everyone benefits from good reviews and the incentive to write them is weak.

A rough norm: review roughly two or three papers for every one you submit, given that each of yours consumes two or three reviews.

And the quality of reviewing in your field is set by what people do, not by what they say about it. The report you write is the norm you are voting for.

The short version

Two reads, separated by time. A report written in one sitting reads as one.

Numbered major and minor points, prioritised by whether they would change the conclusion.

Say what works, so the author knows what to protect.

Recommend consistently with your comments, and be honest rather than generous.

And decline promptly if you cannot do it properly — that is a service, not a failure.

For an independent authoritative reference, consult COPE guidance for publication-ethics guidance for editors and reviewers.