Blueshoe Labs
Instrumented environments that measure how AI agents behave under incentive.
We build the environment, run fixed-specification agents through it, and compare what they do to what people do. Every finding comes with the apparatus to re-run it.
WHY WE BUILT IT
Legal advice is high-stakes decision support. That is what needs measuring.
When someone comes to Blueshoe, they are deciding something consequential: settle, sign, or walk. AI systems increasingly sit inside that moment. The question is not whether a model scores well on a benchmark. It is whether it moves people toward decisions they would not otherwise have made.
Accuracy does not detect deference, over-persuasion, hidden preference, or the drift of an agent that has learned what its evaluator rewards. Nothing we could buy measured those things. We built the apparatus ourselves, on our own production agents, and Labs is that capability turned outward.
A system can be correct and still be a bad advisor.
WHAT WE HAVE BUILT
Three things a reviewer can ask to see on a call.
01
A production agent environment
The Blueshoe platform runs autonomous agents on live legal matters. Every action is logged with its inputs. Every finding is cited to the record it came from. Every outcome is scored and stored as a reward signal against the run that produced it.
run 7f3a… matter M-[redacted] agent intake-screen@2.4 t+00.00 read intake/form.json cite intake:1-42 t+00.31 tool records.request provider=[redacted] status=sent t+04.12 tool records.ingest pages=212 hash=9c1e… t+04.90 infer chronology.build gaps=2 cite records:88,131 t+05.02 flag deadline notice_of_claim due=[redacted] basis=ARS 12-821.01 t+05.40 draft demand.v1 → attorney review (hold) outcome: attorney edited 3 of 41 paragraphs; deadline confirmed; reward +0.82
02
A reward-driven evaluation loop
Agent decisions are scored against what happened next: the attorney's edits, the deadline that held, the client's confirmation. Model versions, prompts, tools, and seeds are pinned per run. A run can be replayed and diffed against its original.
manifest: agent: intake-screen@2.4 spec_hash: a41c… model: provider/model-name@2026-09-14 temperature: 0 tools: [records.request@1.2, chronology.build@3.0] seed: 20261001 env: legal-intake-v3 baseline: human-2025Q2 replay 2026-10-12 vs original 2026-10-01: identical steps: 46/47 diverged at t+05.02: deadline basis cited 12-821.01 (orig) vs 12-542 (replay) verdict: nondeterminism in tool output order; pinned, re-run clean
03
Behavioral benchmarking against people
We run the same fixed-specification agent on several models and measure a named behavior: deference, anchoring, over-confidence, persuasion. Results are reported with confidence intervals next to a human baseline from the experimental literature.
Deference rate (share of trials where the agent changed a correct answer after an authority prompt). Synthetic data for illustration. Real runs report n, the seed set, and the exact model versions in the manifest.
WHAT WE DO
What we do, in plain terms
We build simulation environments with incentives, scarcity, changing information, and competing agents. We run fixed-specification agents across multiple model providers, open-weight and API, with versions, prompts, tools, and seeds pinned. We compute behavioral metrics against published human baselines from experimental economics and decision science. We report effect sizes and AUROC with confidence intervals. We deliver the environment, the runs, and the analysis, so a third party can reproduce every number.
BENCHMARK
Blueshoe Advisory Bench
A public benchmark for how agents advise, not just what they know.
Legal benchmarks today score answers. Advisory Bench scores behavior: what an agent does when a client pushes back, when a counterparty anchors, when the honest answer is uncertain. Tasks are drawn from consumer legal work, graded on rubrics written by licensed Arizona attorneys, and reported against human baselines.
Deference
Does the agent abandon a correct position when a user or an authority asserts the wrong one?
Anchoring
Does a number the other side names first move the agent's valuation, and by how much?
Over-persuasion
Does the agent present uncertain advice with more confidence than the record supports?
Calibration
When the agent says it is 80 percent sure, is it right about 80 percent of the time?
How it is built
- Tasks come from real advisory situations in employment, injury, consumer, and property matters, rewritten as synthetic matters so no client file is involved.
- Each task carries a rubric written by a licensed attorney and a human baseline, from the published literature where it exists and from our own participant studies where it does not.
- Every run is pinned (model snapshot, prompt, tools, seed) and replayable. Results are effect sizes and AUROC with confidence intervals.
Release
The environment, the grader, and the run manifests will be published with the first results, so any lab can run its own models and any reviewer can reproduce ours. A public leaderboard follows the first release.
In preparation. First public results and the open release are scheduled for the current quarter; the chart above shows the output format on synthetic data.
WHO WE WORK WITH
What an engagement looks like
Three shapes we build to. Each ends with a reproducible report and the tooling to re-run it.
Pre-deployment evaluation of a customer-facing agent
An intake or advisory agent, tested for deference, over-persuasion, and calibration against a human baseline before it meets customers. Four to eight weeks. Deliverables: the report, the environment, the replay tooling.
Multi-model behavioral comparison
One fixed agent specification, several model providers, a named set of behaviors. Effect sizes with confidence intervals, and the manifest for every run. Six to ten weeks.
Environment and data program for a model developer
A standing environment plus a graded dataset, refreshed on a cadence, delivered with source and documentation sufficient for the lab's own team to operate it.
Government: we build to government-operable standards. The environment is delivered with source, pinned versions, and documentation sufficient for a third party to reproduce every finding without us in the room.
HOW WE WORK
Four commitments, held on every engagement
Black box by default
We evaluate from inputs and outputs alone. No weights, no privileged access, no cooperation required from the system's owner. A method that needs the door opened only works on systems whose owners open it.
Measured against people
“Is this good?” is unanswerable. “How does this differ from what a person would do, and in which direction?” is a question with an answer. We situate machine behavior against human baselines drawn from decades of experimental work.
Reproducible or it didn't happen
Models change underneath you without notice. An evaluation that cannot name the exact version that produced a result cannot be defended six months later, so ours always can.
Independent of the outcome
We do not build or sell the models we assess, and we hold every system under test to the same protocol. An evaluator with a stake in the result is a marketing department.
Client data stays client data
Benchmarks and evaluations are built on synthetic matters and attorney-graded outputs. Nothing from a client matter is used without the client's informed written consent and de-identification. The duty of confidentiality (ER 1.6) comes before any research interest.
TEAM
Kai Yee Wan
Co-founder and CTO. Architect of the platform.
M.S. EECS, MIT, with research at CSAIL. Embedded systems at Google (Chrome, Nest) and Qualcomm. Adjunct faculty at Northeastern University. Designed the agent runtime, the logging and reward pipeline, and the replay tooling the Lab runs on.
Casey O'Grady
Co-founder and CEO.
Harvard Law School; B.A. in mathematical economics, Rice University. Previously Altman Solon, Lazard, and Osborn Maledon. Leads engagement design and the human-baseline work.
NOTES
Short technical notes on how we measure.
- October 19, 2026Why static benchmarks miss behavior under incentiveA model that answers a question correctly on a test set can still defer, anchor, or over-persuade when something is at stake. The test never put anything at stake.Read
- October 19, 2026Calibrating an environment before measuring agents in itAn environment that has not been run on people, or on a known agent, cannot tell you anything about an unknown one. Calibration is the step most evaluations skip.Read
- October 19, 2026Reproducibility practice: pinned versions, seeds, and replayA finding about a model is only as durable as the ability to re-run it. Here is what we pin, what we record, and how a replay is diffed against the original.Read