Reproducibility as a Working Practice, Not a Virtue

Reproducibility is usually presented as something owed to the field: a standard to meet, a requirement to satisfy, work added at the end of a project.

Framed that way it is a cost, and it gets done badly and late. For a concrete operational example of how a workplace tool approaches a related coordination problem, see here.

Framed as a working method it is different. Most of the benefit accrues to you, during the project, and the version deposited at the end is a by-product rather than an additional task.

The distinction worth keeping

Reproducible: someone can take your data and code and get your numbers.

Replicable: someone can run a new study and get the same conclusion.

The first is a documentation problem and it is entirely within your control. The second is a property of the world.

This article is about the first, which is the one people conflate with the second and then treat as unachievable.

The self-interested case

You are the most frequent user of your own materials. Returning to a project after a year, you are a stranger to it. Everything built for a stranger was built for you.

Revisions become straightforward. A reviewer asking for an additional specification is twenty minutes when the pipeline runs end to end, and a week when it exists as remembered steps performed in a particular order on a particular machine.

Errors are found earlier. The process of making something run cleanly finds mistakes, reliably. Finding them before publication is much better than after.

Collaborators can contribute without asking you to explain the folder structure.

It survives people leaving. When the person who ran the analysis moves on, what remains is what was written down. See a research record that survives you leaving.

What actually breaks reproducibility

Rarely the analysis. Almost always the steps around it.

Manual steps. A spreadsheet edit, a value corrected by hand, a file renamed. These are invisible in the code and they are where most reproduction attempts fail.

If it happened by hand, it did not happen reproducibly. The fix is to do it in code, however trivial it seems.

Undocumented decisions. Which observations were dropped, which outlier rule, why the measure changed in month four.

Environment drift. Code that ran under one version of a package and does not under the next.

Paths that only exist on your machine.

Randomness without a seed.

Files that have been overwritten, so the version that produced the published table no longer exists.

The practices, in order of return

1. Everything in code, from raw data to final table. No manual steps anywhere. This is the single largest change and it is the one people resist because the manual step is faster once.

2. One script that runs the whole thing. From raw input to every number in the paper. If it cannot be run in one command, there is an undocumented step somewhere.

3. Set a seed for anything stochastic.

4. Record the environment. Versions of everything, in a file, generated automatically.

5. Version control, in fields where code is involved. Not for the collaboration features — for the ability to know what produced the published result.

6. Numbers in the paper generated by the code, not typed. Typed numbers become wrong during revision and nobody notices.

7. A decision log, dated, with reasons. The part that cannot be reconstructed later.

The test

Can a colleague reproduce your main result on a clean machine, from what you have, without asking you anything?

Ask one to try, before submission. An hour of their time finds what no amount of self-review will, because your own assumptions are invisible to you.

Whatever they get stuck on is what is missing. Usually: an undocumented step, a path, a version, or a decision that lived in your head.

See sharing data and materials.

When the data cannot be shared

Reproducibility and openness are different things, and this is frequently confused.

Confidential data does not prevent a reproducible workflow. The code, the decisions, the environment and the documentation can all exist, and someone with authorised access can then reproduce the result.

Provide synthetic data with the same structure, so the pipeline can be run and checked even without the real values.

Say precisely what the access conditions are.

The internal benefit is unchanged. Your future self and your collaborators still need the pipeline to run.

Starting mid-project

Most people are not starting from scratch.

Do not attempt to retrofit everything. Start where you are.

Begin with the analysis from raw data forward, and accept that the earliest cleaning may be partly undocumented. Say so.

Write down the decisions you remember, now, before they fade further.

Rebuild the pipeline once, end to end, and fix what breaks. This takes a day and it is usually the day that finds an error.

And apply it fully to the next project, where the cost is a fraction of the retrofit.

What it costs

Hours if planned from the start. Days if retrofitted. The cost is a function of when you decided.

A learning curve for version control and environment management, in fields where these are not standard. Real, and it is a one-time cost that pays back across every subsequent project.

Nothing else. The claim that reproducible workflows are slower does not survive contact with a revision, a co-author joining, or a return to the project after a year.

For supervisors and groups

Set it as the working standard from day one, not as a requirement at submission.

A student who learns this at the start never experiences it as an imposition, and one who is asked to retrofit it at thesis submission will resent it and do it badly.

Provide the template. A project structure, a script skeleton, an environment file. Most of the barrier is not knowing how to start.

See supervision.

The short version

Reproducible is a documentation problem you control; replicable is a property of the world. Do not conflate them.

Manual steps are where reproduction fails. If it happened by hand, it did not happen reproducibly.

One script from raw data to every number in the paper, with a seed and a recorded environment.

Test it by having a colleague run it on a clean machine without asking you anything.

And the main beneficiary is you, eighteen months from now, when you are a stranger to this project.

For an independent authoritative reference, consult the Open Science Framework for infrastructure for transparent research practice.