Guides

A second opinion on another tool's scores

Your eval score went up. Is it real? And can you trust the judge that measured it? If you already evaluate with another tool, export its per-case results before and after a change, and Oloproof answers both questions on them without running anything again.

What it reads

One file per system version, CSV or JSON Lines, with one row per case and criterion:

Column	What it holds
case_id	The case's id. Two files pair only on an id both contain
criterion	The metric's name, which becomes the metric a rule names
score	A number on the metric's scale, or empty if the tool did not score that case
passed	true or false instead of a score, or empty
score_min, score_max	The scale a score lies on (0 and 1 for most tools), required for a score
judge, judge_version	The model that judged, if a model did. Optional

Nothing else is read, and a file with any other column is refused: a file holds case ids and scores, never a question, an answer or any other content. The recipes below turn each tool's results into this file.

Run it

oloproof init --example second-opinion second-opinion
cd second-opinion

The example holds two exports of a support assistant's faithfulness on 150 questions, before and after a prompt change, as another tool's judge scored them: a score from 0 to 1, and its pass or fail at 0.7. make_exports.py is the simulation that wrote them. The average score went up by about 0.02.

oloproof scores import before.csv after.csv --policy release.yaml

faithfulness: +0.023 [-0.024, +0.068] · 150 paired · 0 missing · 0 excluded
faithfulness_passed: +5.3 points [-6.9, +18.0] · 150 paired · 0 missing · 0 excluded
Decisions
  score-went-up  faithfulness  superiority  INSUFFICIENT_EVIDENCE  evaluator_not_validated
  more-answers-pass  faithfulness_passed  superiority  INSUFFICIENT_EVIDENCE  evaluator_not_validated
Gate: BLOCK (exit 3)
score-went-up, more-answers-pass: the judge (or model or custom evaluator) behind these rules has not been measured against people yet, so it may not decide.

Each file became a run, and the two runs were compared case by case. The answer has two halves.

Is it real? Not yet. The score rose by 0.023, but the interval runs from 0.024 lower to 0.068 higher: with 150 questions, a change this small cannot be told apart from noise. The share of answers the judge passes rose 5.3 points, with an interval from 6.9 points lower to 18.0 higher.
Can you trust the judge? Nobody has checked it. release.yaml lets a judge decide a release only once people's labels have measured it (require_validated_evaluators: true), so both rules stop at evaluator_not_validated before they reach the interval.

To see the first half alone, as if the judge had been checked:

oloproof scores import before.csv after.csv --policy statistics-only.yaml

  score-went-up  faithfulness  superiority  INSUFFICIENT_EVIDENCE  interval_overlaps_zero
  more-answers-pass  faithfulness_passed  superiority  INSUFFICIENT_EVIDENCE  interval_overlaps_zero

In CI, add --summary markdown >> "$GITHUB_STEP_SUMMARY": the pull request then opens on "Is it real?" and one line per rule, saying "not established" and why, and the exit code is the same. The import's own lines go to standard error, so the summary receives the answer alone (CI).

Without --policy the command reports the comparison and decides nothing. With one file it reports that run, and --policy gates it on the policy's run rules.

Checking the judge

People label a random sample of an imported run's cases, blind to the judge, and Oloproof measures the judge against them:

oloproof labels export RUN_ID --criterion faithfulness_passed --sample 60 --out sample.csv
oloproof labels import sample.csv
oloproof evaluators validate EVALUATOR_ID --by NAME

The sample shows each case's id and nothing of the judge's verdict. The labeller reads the case in the tool that holds it, by its id, and writes pass or fail in passed. oloproof evaluators list gives the evaluator's id. Once validated, the rules that read that judge decide on the evidence alone.

Labels are pass or fail, so they check a judge's pass or fail verdict, not its score. Import the verdict beside the score, as the example does with faithfulness_passed: that is the criterion people can validate.

From your tool

The example's recipes/ folder holds one short script per tool. Each reads that tool's results and writes the file, keeping ids and scores only. Run one per version, then import the two files.

Tool	Recipe	What identifies a case across two runs	Notes
DeepEval	deepeval_results.py RESULTS_JSON OUT	The test case's name	Save results with DisplayConfig(results_folder=...) and name every LLMTestCase. Each metric gives its score and its pass or fail as <metric>_passed, with the judge model
Braintrust	braintrust_experiment.py PROJECT EXPERIMENT OUT	The dataset record the experiment ran (origin.id)	Scores from 0 to 1; --judge names the model if one judged them all
LangSmith	langsmith_experiment.py EXPERIMENT OUT --scale LOW HIGH	The dataset example (reference_example_id)	A true or false score becomes pass or fail; a number needs its scale
Langfuse	langfuse_experiment.py EXPERIMENT SINCE OUT	The dataset item (experiment_item_id)	Langfuse 5's experiments; numeric scores on their score config's range, or --scale

The recipes follow each tool's SDK as published on 16 October 2026 (DeepEval 4.3.1, braintrust 0.46.1, langsmith 0.14.11, langfuse 5.0.2) and are tested on results of that shape, not against a live account. A tool that renames a field breaks its recipe, not the import: the file it writes is the contract.

What a file must get right

A case present in one file only is missing on the other side, counted and bounded at its worst; it is never paired with another case.
A second row for one case and criterion, a score outside score_min and score_max, a row with both a score and a pass or fail, and a criterion that is a score in one file and pass or fail in the other are refused, naming the row.
Files compared must agree on each criterion's scale and judge: scores from two judges are two instruments, and their difference would be partly the instruments'.
Each file is a system version identified by its contents. Importing the same file again reuses what was stored; nothing is ever executed, so no rerun can reproduce or change an imported score.
--tool records where the scores came from, and --name names each file's system; both are kept with the run.

What is not built

Reading a tool's own export format directly: a recipe turns it into the file above.
Uploading the files to the hosted site without installing anything.
Validating a judge's scores, as opposed to its pass or fail verdicts.

Where to go next

Comparing a candidate to a baseline: what a paired comparison reads.
Comparison rules: superiority, non-inferiority and equivalence.
Judges: how a judge is measured against people before it may gate.