Reproducibility practice: pinned versions, seeds, and replay

Models change under you. A provider ships a new snapshot, a tool returns its fields in a different order, a prompt template gets a word changed. Six months after an evaluation, someone asks whether the finding still holds. The honest answer is unavailable unless the run can be reproduced exactly as it happened.

We treat every run as an artifact with a manifest. The manifest names the agent specification and its hash. It names the model and the dated snapshot it was called at, and the sampling parameters. It names every tool and the environment, each with a version. It names the seed set and the human baseline the results are compared to. If any of those is missing, the run is not reportable.

Pinning the model version is the step people underestimate. For API models we record the provider's dated snapshot identifier and refuse to run against an alias that can change. For open-weight models we record the weights hash and the inference stack version. When a provider retires a snapshot, the finding is marked as no longer reproducible rather than silently re-run on a successor.

Seeds cover the randomness we control. Sampling temperature is zero unless the behavior under study requires otherwise, and when it does, the seed set is fixed and stated. Order effects are handled by running each condition in several orders and reporting the spread, not by hoping they average out.

Replay is how we check our own work. A replay re-executes the run from the manifest and compares it step by step to the original log. Same tool calls with the same inputs, same intermediate outputs, same final decision. Any divergence is reported with the step at which it occurred and the first differing value.

Divergence is informative. The common causes are tool nondeterminism, such as a records service returning pages in a different order. Or an unpinned dependency in the environment. Or provider-side changes behind a stable model name. Each has a fix, and the replay diff is what finds it. A clean replay is the evidence that the pinning actually worked.

The deliverable for every engagement includes the manifests, the logs, the replay tool, and the analysis code. The documentation is written for a team that has never spoken to us. The test is simple. Could a reviewer at the client, or at a government program, reproduce the headline number from the delivered materials alone? If not, the engagement is not done.

This is more work than producing a chart. It is also the only way a finding about model behavior survives the next model release. And it is the only basis for a deployment decision.