Reporting Results You Did Not Expect

The analysis is done and the answer is not the one you expected. Nothing, or the opposite, or an inconsistent pattern across specifications.

This is a normal outcome of research and it is reported badly more often than any other kind, for reasons that are structural rather than personal: the incentives favour clean stories, and a messy result is harder to publish. For a concrete operational example of how a workplace tool approaches a related coordination problem, see learn more.

The temptations, named

Worth listing because they arrive as reasonable thoughts rather than as misconduct.

Reframing the hypothesis to match the result. Writing the paper as though you had predicted this all along. It makes the paper cleaner and it misrepresents what happened, and it inflates how much the finding should update anyone.

Searching specifications until one works. Trying twenty analyses and reporting the one that reached significance. Each individual choice feels defensible; the aggregate is not.

Splitting the sample until a subgroup shows the effect. With enough subgroups, one will.

Dropping the inconvenient outcome from the paper.

Reporting it as "trending" when it is not.

None of these require bad intent. They are what happens when a null result is harder to publish than a positive one, and the person doing the analysis knows it.

What to do instead

Report what you predicted, and what you found. Both. If they differ, that is the interesting part.

Distinguish confirmatory from exploratory analysis, explicitly. The analysis you planned is one thing; the analysis you did afterwards because the first was surprising is another. Both are legitimate; conflating them is not.

Say how many specifications you ran and show them, in an appendix if not in the main text. A specification curve or a table of alternatives is more convincing than a single clean result.

If you preregistered, report against it. Including the deviations, with reasons.

And say plainly that the result was unexpected. It is more honest and, in most fields, better received than a paper pretending otherwise.

Making a null result publishable

It is harder. It is not impossible, and the difference is mostly in the framing.

Establish that the test was capable. A null from an underpowered study says nothing about the world. A null from a well-powered study says something. This is the whole argument — if you cannot show the design could have detected an effect of the size that matters, the paper will not survive review.

Report the interval, not just the absence of significance. "The effect is between -0.02 and 0.03" is a finding: the effect, if any, is small. "Not significant" is not.

Say what would have been detected. "We could have detected an effect of 4% with 80% power; the point estimate is 0.3%."

Frame it against a specific claim. A null is publishable when it contradicts something the field believed, and much less so when nobody expected an effect.

Rule out the boring explanations. A failed manipulation, a broken measure, a sample that was wrong. If those are not addressed, reviewers will assume one of them.

Reporting an inconsistent pattern

Harder than a null and more common.

Show the inconsistency. Do not select the specifications that agree.

Say what would explain it, as hypotheses rather than as conclusions. Different populations, different measures, different timing.

Do not manufacture a mechanism. A post-hoc story that accounts for the pattern is a hypothesis, and presenting it as a finding is the most common way messy results become misleading papers.

Be clear about what you now believe and how confident you are. Readers can handle "we do not know why this differs"; they cannot handle a confident explanation that turns out to be invented.

The framing that works

Lead with the question, not the expectation. A paper framed as "we tested whether X" accommodates any result. One framed as "X causes Y" has nowhere to go when it does not.

Write the introduction after the analysis. See the first paragraph. A paper introduced as though the result were predicted was frequently introduced before the result existed.

Present the unexpected result as information, which it is. The field's belief was untested or wrong, and now there is evidence.

Be modest about what one study establishes. A single unexpected result is a data point, not an overturning.

Where to send it

Some venues publish null and negative results explicitly, and the number is growing. Look for them.

Registered reports, where acceptance is decided on the design before results exist, remove the problem entirely. If your field has them and the study is not yet run, this is worth considering at the design stage rather than at the writing stage.

Preprints get the result into circulation regardless. See preprints.

And the file drawer is the worst option. A result that never appears distorts the field's evidence base and, if someone runs the same study, wastes their time as well as yours.

For reviewers and supervisors

Since the incentives are collective.

Do not require a clean story. A reviewer who rejects for messiness is part of the reason messiness gets hidden.

Ask about power on any null, and accept a well-powered one.

Ask how many specifications were run, on any result — not only the surprising ones.

Supervisors: say out loud that most research does not work. A student who believes a null is a failure will eventually be tempted, and hearing otherwise from someone experienced is what prevents it. See supervision.

The short version

Report what you predicted and what you found, and say when they differ.

Separate confirmatory from exploratory, explicitly, and show the specifications you ran.

A null is publishable if the test was capable — show the power and report the interval, not the absence of significance.

Do not manufacture a mechanism for a messy pattern. A post-hoc story is a hypothesis.

And the file drawer is the worst option, for the field and for the next person to run it.

For an independent authoritative reference, consult the EQUATOR Network for reporting guidance for research writing.