One change, end to end

Your score went up after a change. This page follows one real change through every question Oloproof asks before it lets that change ship: is the judge trustworthy, does the change meet the bar on its own, is it better than what it replaces by the judge and by people, and what to try next. The system is a small support bot answering from 40 documents, run on your machine with local models; every number below is what the engine printed. Copy it from examples/one_change and run REVIEWER=yourname ./run_demo.sh.

The change

The baseline prompt tells the bot to keep the exact figures, names and units. The candidate asks for one or two warm, natural sentences. Both are run on the same 120 questions, with a deterministic check (answer_correct, does the answer contain the expected fact) and a local model judge (judge_correct, does it answer with the same fact):

oloproof run --policy release.yaml
oloproof run --config candidate.yaml --policy release.yaml

Answers right went from 80.0% to 82.5%, the judge's pass rate from 82.5% to 83.3%. That is the apparent improvement. Neither run decides anything yet: the release rule reads the judge, and the judge has not been checked, so the rule says INSUFFICIENT_EVIDENCE with evaluator_not_validated.

Can the judge be trusted?

People label a blind random sample of the candidate's answers, without the judge's verdict, and the engine measures the judge against them:

oloproof review CANDIDATE --criterion judge_correct --by you

In the recorded session the judge agreed on 77 of 79 labelled cases (97.5%, low end 91.2%) with a bias of -2.5 points, inside the default bars (agreement 0.70, bias 0.35), so it was validated and may decide. In that session the labels were given by an agent standing in for a reviewer, which shows the workflow and is not a person's judgment; when you run it, you are the reviewer.

Does the candidate meet the bar on its own?

release.yaml asks for a judge pass rate of at least 80%:

oloproof gate CANDIDATE --policy release.yaml --config candidate.yaml

83.3% with a range of 75.4% to 89.6% straddles 80%, so the rule is INSUFFICIENT_EVIDENCE, and the engine says about 592 cases would decide it. A trusted judge does not make a close result clear.

Is it better than the baseline?

compare.yaml asks two questions. Is the candidate better by the judge, kept only if people show the judge errs alike on both prompts (judge_bias_bar: 0.10)? And by people's own labels on both answers to the same cases (people_pass_diff, a human_difference metric), is it no worse than 10 points below?

oloproof review CANDIDATE --against BASELINE --criterion judge_correct --by you --sample 40
oloproof compare CANDIDATE BASELINE --policy compare.yaml --config candidate.yaml

The review shows both answers as A and B, in an order set by a coin, without naming the runs. In the recorded session the judge's difference was +0.8 points (-5.1 to +7.3): INSUFFICIENT_EVIDENCE, and about 3,690 paired cases would be needed to decide it. People passed the candidate 36 times and the baseline 35 times in 40 pairs: +2.5 points (-11.1 to +17.7), so "no worse than 10 points" was INSUFFICIENT_EVIDENCE too. The gate blocked. The change was a style change the scores barely noticed, and 120 cases cannot tell a 1-point change from none.

Why does it fail where it fails?

oloproof diagnose CANDIDATE --intervention gold-context --criterion answer_correct --config candidate.yaml

The engine re-ran the 21 failures with the right passage handed to the model, and again without it as a control. The control recovered none; the right passage recovered 14: 7 where retrieval never found the passage, 7 where it was found but ranked below the top 3, and 7 that failed anyway. It proposed a deeper top_k as a hypothesis, then tested it on the same cases:

oloproof diagnose CANDIDATE --intervention top-k --top-k 8 --criterion answer_correct --config candidate.yaml

All 7 ranked-out cases recovered, with a control that recovered none.

The follow-up

followup.yaml is the candidate with top_k: 8, a new system version run in full:

oloproof run --config followup.yaml --policy release.yaml
oloproof compare FOLLOWUP CANDIDATE --policy followup_compare.yaml --config followup.yaml

Its judge pass rate was 90.0% (83.1% to 94.8%), so the 80% floor was PASS. Against the candidate, more answers right (+3.3 points) was still INSUFFICIENT_EVIDENCE, and the judge's pass rate no worse by more than 5 points was PASS. The judge was checked on the candidate's answers, not the follow-up's, and nobody labelled a pair of the follow-up, so people's difference there decides nothing.

What this cannot tell you

One small bot, one change, 120 cases and local models. It shows how the engine reaches each decision on a real change; it says nothing about another system. A local sample is good faith: only a sample a workspace draws is checked there. The full record, with every run id, is docs/audits/DOGFOOD_LARKSPUR_2026-10-26.md. See Judges, Comparison rules, RAG and Limits.