Calibrating an environment before measuring agents in it

Suppose you build an environment to measure deference and run a model through it. It changes its answer under pressure forty percent of the time. Is that a lot? The number means nothing on its own. It needs a reference: what a person does in the same environment, and what a trivially deferential or trivially stubborn agent would score.

Calibration is the work of establishing those references before the model under test ever enters the room. It has three parts, and each one catches a different kind of mistake.

The first is the human baseline. Where the behavior has been studied in people, the published results give an expected range. Where it has not, a small human study in the same environment does. Either way the baseline is a line on the chart, not a footnote. The question being answered is how the machine differs from the person.

The second is the pair of degenerate agents. One always defers; one never does. Running both through the environment proves that the instrument can see the behavior at all. If the always-deferential agent scores sixty percent instead of a hundred, the environment is measuring something other than deference. The number for the real model is then uninterpretable.

The third is sensitivity. Change the thing the environment is supposed to be sensitive to, such as the authority of the voice applying pressure, and confirm the measurement moves. Change something it should ignore, such as the order of unrelated facts, and confirm it does not. An environment that responds to the wrong variables produces findings about those variables, dressed as findings about the model.

Only after those three steps does the model under test go in. Its score is reported as a distance from the human baseline, with an interval. The degenerate scores are shown as the ends of the scale. A reader can see at a glance whether the environment could have found the behavior and whether the model showed it.

Calibration also fixes a quieter problem: the drift of the environment itself. Prompts get edited, tool outputs change format, a data source updates. Re-running the degenerate agents and the sensitivity checks after any change is the cheapest way to know the instrument still reads true. We run them on every release of an environment, the way a lab re-zeroes a scale.

None of this is novel. It is what experimental scientists do before trusting an apparatus. It is skipped in AI evaluation mainly because benchmarks arrived first and made measurement look like scoring. An environment is not a test set. It is an instrument, and instruments are calibrated.