Evidence for every AI decision.

Know if your AI system is good enough.

Your eval score went up. Oloproof tells you if it's real, and if the judge can be trusted. Works on your own runs or on scores from the tool you already use.

Start evaluating See an evaluation

$ pip install oloproof$ oloproof scores import before.csv after.csv

Or the hosted second opinion, with nothing to install.

Python. The engine, CLI and SDK are free, Apache-2.0 licensed. Runs locally; nothing is sent.

oloproof / run · live-rag

SYSTEM

RAG bot · local model · no baseline

REQUIREMENT

answer_correct ≥ 95%

DATASET

120 questions · a look every 10 cases

CASES RUN	ESTIMATE	95% ANYTIME-VALID	STATE
10 / 120	70.0%	[1.0%, 100.0%]	Insufficient evidence
20 / 120	80.0%	[4.9%, 99.8%]	Insufficient evidence
30 / 120	80.0%	[9.5%, 99.3%]	Insufficient evidence
40 / 120	82.5%	[15.3%, 99.0%]	Insufficient evidence
50 / 120	84.0%	[21.6%, 98.7%]	Insufficient evidence
60 / 120	81.7%	[26.8%, 97.5%]	Insufficient evidence
70 / 120	82.9%	[33.7%, 97.0%]	Insufficient evidence
80 / 120	82.5%	[40.2%, 96.1%]	Insufficient evidence
90 / 120	80.0%	[45.2%, 94.0%]	Failing

Each interval is an anytime-valid 95% interval, which stays valid however often it is checked, over all 120 cases, with every case not yet run entered as a failure for the lower bound and as correct for the upper one. That makes it wider than a count: at 40 cases, 33 correct plus all 80 not yet run counted correct is 94.2%, but the upper bound, which also allows for chance in the 40 already run, reads 99.0%. The floor is held against that upper bound, not the estimate: at 90 cases it fell to 94.0%.

BASELINE

docs assistant · rerank off

CANDIDATE

docs assistant · rerank on

DATASET

golden panel · 58 cases

METRIC	BASELINE	CANDIDATE	PAIRED Δ	95% CI	STATUS
	0.618	0.745	+6.2	[-36.3, +45.5]	? Inconclusive
48 cases were measured in both versions and 10 in only one. The difference of +6.2 points has an interval from -36.3 to +45.5: it crosses zero, so this panel cannot tell a gain from a loss.
	0.683	0.742	+3.7	[-73.9, +76.2]	? Inconclusive
Only 27 cases were measured in both versions and 31 in only one. The missing cases are priced into the interval rather than dropped, which is why it runs from -73.9 to +76.2.
	0.685	0.760	+6.5	[-39.4, +48.4]	? Inconclusive
46 cases were measured in both versions and 12 in only one. The difference of +6.5 points has an interval from -39.4 to +48.4, which crosses zero.
	55 / 58	51 / 58	4 fewer	n/a	! Bounded
The reranked version was observed on 4 fewer cases than the other. A case that could not be measured is bounded inside the interval, never dropped.
	41 / 58	31 / 58	10 fewer	n/a	! Bounded
The reranked version was observed on 10 fewer cases. At this cutoff more than half the panel could not be measured in both versions, and the interval says so.
	54 / 58	50 / 58	4 fewer	n/a	! Bounded
The reranked version was observed on 4 fewer cases. Each missing case widens the interval by what it might have been.

EACH VERSION ALONE

Each version measured on its own, with its interval and the cases behind it. Read alone, the two intervals overlap on every metric; the paired comparison is the one that decides.

page_hit_1

rerank off0.618 [0.449, 0.760]55 / 58 observed

rerank on0.745 [0.519, 0.875]51 / 58 observed

page_hit_3

rerank off0.683 [0.350, 0.875]41 / 58 observed

rerank on0.742 [0.270, 0.939]31 / 58 observed

answer_correct

rerank off0.685 [0.501, 0.819]54 / 58 observed

rerank on0.760 [0.519, 0.888]50 / 58 observed

Pipeline calls: 116
Judge calls: about 110
Wall clock: about two minutes
Golden panel: 58 questions, unchanged

INTERVENTION · GOLD CONTEXT

On a staged rerun of the same panel, 23 answers failed answer_correct. Each was generated again with the gold passage in place of what retrieval returned.

15 recovered with the gold passage8 did not

RETRIEVAL_MISS5

GENERATION_FAILURE7

UNRESOLVED11

THE RECOMMENDATION, AS RECORDED

retrieval coverage: RETRIEVAL_MISS in 5 of 15 recovered cases; try depth, index_configuration

A control arm re-ran the same cases unchanged, and a case that passed there too was left unresolved, so no recovery it counts is the pipeline's own variation. The diagnosis declared what it assumed:

claim_limited_to_selected_casesretrieval_deterministiccriterion_evaluator_unvalidated

THE BUNDLE

The run, every case behind its numbers and every diagnosis, as files you keep.

$ oloproof export RUN_ID

run.jsonthe run, its suite, system and evaluators
cases.jsonlevery case: input, output, judgments, digests
diagnoses.jsonlinterventions and what they recovered
signoffs.jsonlwho shipped against a blocked gate, and why

$ oloproof gate RUN_ID --policy release.yaml

Re-decides the stored run under a policy, without rerunning the system or its evaluators.

MEASURED AGAIN, THREE TIMES PER CASE

About one observed case in nine changed its verdict between identical measurements: 6 of 54 on answer_correct, 7 of 57 on page_hit_1. A single run of this panel does not have four decimal places of resolution.

DECISION

Failing

Even with all 30 cases not yet run counted as correct, the anytime-valid upper bound fell to 94.0%, under the 95% bar. So the run stopped, and spent nothing on them.

EVIDENCE

Cases in the suite120

Cases run90

Not needed30

Exit code1

FOUR QUESTIONS, ONE RECORD

What a reviewer, a risk owner and a buyer each ask of a release.

Each answer below is what the engine printed for one run of the assurance demonstration: whole lines, exactly as printed, with only the lines about other runs left out.

examples/assurance_demo

Know how your AI performs.

Every rule is decided on an interval: pass, fail, or not enough evidence yet, with how many more cases would settle it. Here, 14 of 15 answers were right, and 15 cases cannot show the bot clears 80%.

$ oloproof gate run_01M56TR6F0BGDWDXTNMCT2X1QG
correct-floor: INSUFFICIENT_EVIDENCE (interval_overlaps_threshold)
  a suite of about 41 cases (26 more) would decide it, if the next cases go like these; at that size it is settled about half the time
Gate: BLOCK (exit 3)

How a rule decides

Find the risks before they matter.

A ceiling bounds how often something goes wrong on the cases your suite holds, and fails however good the rest is. Oloproof measures the ceilings your policy declares, writes seeded attack cases for a sandbox, and reads a red-team tool's grades when you have them.

$ oloproof gate run_01M56TR6ZVGA44Q7V2M00Y41JR
correct-floor: PASS (lower_bound_meets_minimum)
no-guarantee: FAIL (observed_failures_exceed_limit)
Gate: BLOCK (exit 1)

Risk ceilings

Turn evidence into accountability.

A passing release can wait for two named approvals. The decisions stay what the evidence said, and the gate says who approved.

$ oloproof gate run_01M56TR70JNNFERFSQ22QPKKNH
correct-floor: PASS (lower_bound_meets_minimum)
no-guarantee: PASS (observed_failures_within_limit)
Every rule allows the release, and this policy requires two approvals: oloproof signoff run_01M56TR70JNNFERFSQ22QPKKNH --by NAME --reason "..." then a second approver with --approve.
Gate: BLOCK (exit 6)

Approving a passing release

Trust requires continuous evidence.

A decision says when the system it describes has changed since. Once the system is live, people review a sample Oloproof draws from real traffic, while the content stays on your side.

$ oloproof gate run_01M56TR72E865651B15590ZJ7V
correct-floor: PASS (lower_bound_meets_minimum)
no-guarantee: PASS (observed_failures_within_limit)
Evidence: stale. Changed since this run: the system (69ea7c33e1e1 decided, 38226f8e36bf now). The decisions describe the project as it was; run it again to decide it as it is.
Gate: ALLOW (exit 0)

Freshness · Production review

WHY EVALUATION

Prompt testing by hand does not survive production.

Teams ship model and prompt changes weekly. Spot-checking a handful of outputs in a playground tells you almost nothing about what changed, for whom, or whether it is safe to release.

Oloproof replaces that with a measured record: the same cases, the same judges, the same seeds, run on every change.

01

Silent regressions

A prompt edit fixes one failure class and quietly breaks another. Nobody notices until a customer does.

02

Unrepeatable results

On a live RAG system, about one observed case in nine changed its verdict between identical measurements.

03

Scores without meaning

A live harness would have scored 6 of 58 cases as zeros: its vendor's server errors and its judge's unparseable verdicts, charged to the model. An average cannot carry a decision.

04

No record to show

When risk, legal, or a customer asks what you tested, screenshots in a thread are not an answer.

THREE LAYERS

What happened, how sure you can be, what happens next.

MEASUREMENT

What happened

Every case, score, latency and cost, traced to the run, the judge and the data that produced it.

EVIDENCE

How sure you can be

An interval behind every decision, and judges checked against people before they are allowed to count.

DECISION

What happens next

Pass, fail, or not enough evidence yet, with how many more cases would settle it and which part broke.

Diagnostics23 failed cases, gold context

Unresolved: gold context did not settle it11 cases

Generation failure: right context, wrong answer7 cases

Retrieval miss: answer reachable, not retrieved5 cases

Rerank on minus off95% interval, points

hit@1

hit@3

correct

Every interval crosses the dashed zero line. Read off the point estimates alone, the reranker would have shipped.

PLATFORM

Built for teams that have to answer for what their AI does.

Evaluation suites as code

Define datasets, judges, and thresholds in version control. Review evaluation changes the way you review code.

Judges you can audit

Programmatic checks, LLM judges, trained classifiers that score text on your own machine, and human labels in one pipeline. Each judge's agreement with people is measured with an interval, and one that people have not checked, or that misses the bar you declare, cannot decide a release.

Regression gates in CI

Fail a pull request when a rule's interval misses its threshold, or when the evidence cannot tell. The comparison counts the cases that changed in each direction.

Trace-level evidence

Every score links to inputs, outputs, tool calls, retrieved context, and judge rationale. Nothing is a black box.

Cost and tail latency

Quality is never the only axis. Track per-case cost and tail latency alongside correctness, and hold a quantile of either to a ceiling on its interval.

Evidence for people who need the decision

Share the run, diagnostics, and recommendation without changing the measured decision it contains.

HOW IT WORKS

Three steps, then it runs itself.

Define the suite

Bring your own cases as a JSONL file. Declare judges and the thresholds a release must clear.

Run on every change

One command locally, the same command in CI. Prompt, model, retrieval, and tool changes all get the same treatment.

Decide with the evidence

Read the decision and its intervals, open the failing traces, and keep the evidence attached to the call you made.

TWO WAYS IN

Evaluate a system

Hold one system to what you require of it, with no baseline needed.

Compare a change

Compare a change with what is live, on the same cases, and know whether the difference is real.

~/olodemo · after oloproof init

$ oloproof run
Run run_01M3B0CCPA0JFQQ7HWRKYNWFJC [DECIDED/COMPLETE]
Gate: ALLOW (exit 0)

  Rule         Metric       State  Reasons
  label-floor  exact_label  PASS   lower_bound_meets_minimum

  Metric       Estimate  Interval          N
  exact_label  100.0%    [88.4%, 100.0%]   30 / 30 observed · 0 missing

Cache: execution 0 hit/30 miss; judgment 0 hit/30 miss

$ oloproof run
Run run_01M3B0CXTTVPTBXAFK3WNGFX42 [DECIDED/COMPLETE]
Gate: ALLOW (exit 0)
Cache: execution 30 hit/0 miss; judgment 0 hit/30 miss

PRODUCTION REVIEW

Once it is doing real work.

Once it is live, people review a sample Oloproof draws from real traffic, in the browser, while the content stays on your side. The workspace decides on what they found, not on what anyone chose to show it.

How production review works

EVIDENCE YOU CAN HAND OVER

Every number traces back to a case.

Runs are immutable and dated. Datasets, prompts, judges, and configuration are versioned together, so any result can be reproduced or challenged.

Immutable

runs, with full lineage

Local

runs on your machine, your evidence stays there

Bounded

missing cases priced, never dropped

Read off the point estimates, the reranker delivers +12.7 points of hit@1. Read honestly, this panel cannot tell whether it helps or hurts.

Oloproof against a live RAG system, 22 September 2026

Evidence bundle

oloproof export RUN_ID

run.jsonthe run, its suite, system and evaluators

cases.jsonlevery case: input, output, judgments, digests

diagnoses.jsonlinterventions and what they recovered

signoffs.jsonlwho shipped against a blocked gate, and why

An exported evidence bundle: the run and every case behind its numbers, as files you keep.

THE SAME DISCIPLINE, POINTED HERE

What this is not, yet.

Everything below is true on the day you are reading it.

01
The hosted deployment is an alpha on a single server, with no uptime commitment.
02
There is no security audit, no SOC 2 report and no DPA. There are no customers and no case studies, so there are none quoted.
03
Notifications are email only: no Slack, pager or webhook. There is no report document. A run is read in the workbench or exported as a bundle.
04
It has been run end to end against exactly one live third-party system: the comparison above. The single run above is a bot built for testing, on a laptop.

Decide on evidence, not on a score.

Bring one system and one dataset. Start with one local run and keep the evidence on your machine.

Start evaluating Read the docs

Prefer we run it with you? Tell us what your system must do, and we will set up the first evaluation alongside you and walk through the evidence. Talk to us