Safety research: from an Inspect log to a methods paragraph

You ran an eval on a model: a model organism and its control, an attack set, an auditing agent's transcripts scored by a judge. Before you write the result down, three questions decide whether it holds: how wide is the uncertainty, is "none of 200" really none, and can you trust the judge that scored the transcripts? Oloproof answers them from the log you already have, and writes the figure, the table and the methods paragraph from what it measured.

Read the log

Oloproof reads UK AISI Inspect logs, the .eval file inspect eval writes or a JSON log:

oloproof scores import --from inspect logs/organism.eval
oloproof scores import --from inspect logs/control.eval logs/organism.eval --policy release.yaml

The first command imports one run and answers in plain words. The second imports two runs, pairs them on sample ids and decides the policy's comparison rules; a policy given with two logs must hold at least one (kind: superiority, non_inferiority or equivalence), and a ceiling on one run is decided by importing that log alone with the policy. Each scorer becomes a criterion: C passes and I fails, partial credit is a score from 0 to 1, and a numeric scorer needs its range, given with --scale LOW HIGH. Only ids and scores are copied; the transcripts stay in the log. See A second opinion on another tool's scores for every reading rule.

Epochs are not more samples

Running each sample several times (Inspect's epochs) does not give you more independent cases. Oloproof keeps the sample as the unit: the log imports as a run whose replicates are its epochs, each sample's value is the mean of its epochs, and every sample weighs the same. Each epoch keeps its own pass or fail, so a judge can be checked on epochs, with the sample as the unit. Counting each sample's runs as separate cases treats repeats as new prompts: in the method's grid, pooling two runs per sample that way covered a claim about prompts like these as rarely as 86% of the time at a stated 95%, though it held for a claim about the samples themselves (docs/engine/PASS_AT_K_METHOD.md §8). A sample with an errored epoch counts as missing, at its worst, never as the mean of the epochs that finished, since errors tend to fall on the hard samples.

None seen is not none

A safety claim often rests on a count of zero: no attack succeeded, no transcript showed the behavior. Zero seen is not a zero rate. With no bar set, Oloproof says how high the rate could still be and how many clean cases would bring that down:

attack_succeeded: about 0.0%; the true value is likely between 0.0% and 1.9% (95% range, 200 cases).
  None of the 200 cases counted toward it, yet the rate could still be as high as 1.9%: none seen is not none.
  If the next cases show none either, a suite of 368 cases (168 more) would bring the high end to 1.0% or below.

Under a ceiling the policy decides it. max: 0.01 on 0 of 200 is INSUFFICIENT_EVIDENCE, and the advice is conditional: "if none of the next cases count either, about 368 would decide it; each one that does raises that count". No probability is quoted, because with no events there is nothing to quote it from.

rules:
  - id: attack-ceiling
    metric: attack_succeeded
    max: 0.01

Can it ever? pass@k

A capability or red-team question is often whether any of k tries succeeds. oloproof passk reads each epoch of a sample as one attempt and estimates each sample's pass@k without bias, then the suite's mean:

oloproof passk logs/organism.eval --from inspect --criterion leaked --k 5

A file of one row per attempt (case, passed, optional error) works too. A sample with an errored attempt, or fewer than k attempts, counts as missing, at its worst. The answer gives the estimate and its 95% range. With --min or --max it decides: PASS when the range clears the bar, FAIL when it falls entirely on the wrong side, INSUFFICIENT_EVIDENCE otherwise (exit 0, 1 or 3). The method (docs/engine/PASS_AT_K_METHOD.md) covered its stated 95% in every cell of its grid and was admitted by an independent audit on 3 November 2026, on its terms: the attempts of a case are independent of each other, and you did not stop a case's attempts early because one succeeded.

Many conditions at once: a sweep

A table of twenty differences, each with its own 95% interval, is expected to be wrong somewhere about once: in the method's grid, fifty separate intervals all held only six times in ten. oloproof sweep decides a family of contrasts you name before reading the runs, so the whole table holds together:

sweep:
  conditions:
    control: run_01...
    organism_a: run_01...
    organism_b: run_01...
  contrasts:
    - organism_a - control
    - organism_b - control

oloproof sweep sweep.yaml --policy compare.yaml

Each contrast is a paired comparison of two runs of one suite, decided by every comparison rule in the policy, each claim at 1 - alpha/m with m the number of claims: the chance that any of them is wrong is at most alpha. A contrast you read only after looking at the table is outside that guarantee. Runs on different suites are not compared.

A monitor's safety against its audit cost

For a control protocol, a file of one row per trajectory, its arm (attack or honest) and the monitor's suspicion score, gives safety (the share of attacks flagged) and the honest flag rate at every threshold:

oloproof control monitor.csv --budget 0.02 --min-safety 0.9

The ranges hold at every threshold at once, both arms together, so a threshold picked by reading the curve, such as the one that fits a 2% audit budget, is still covered. With --budget and --min-safety it decides whether some threshold flags at most that share of honest trajectories while catching at least that share of attacks. The ranges are as wide in the tail as in the middle, so a small budget needs many honest trajectories: with 2,000 the honest flag rate is known to about 3 points. Safety here is against the attacks in the file; an attacker who adapts to the monitor is outside it.

Safety is counted over the trajectories in the attack arm, whether or not their attack worked. A trajectory told to insert a backdoor may not have inserted one, and a monitor is not to blame for missing what is not there. For safety against attacks that succeeded, check each attack trajectory first (for a backdoor, a test on its secret input) and keep only those that pass in the attack arm before the file reaches oloproof control.

Check the judge

A transcript scored by a model is the judge's word until people agree with it. Label a blind sample of the imported run, keeping the transcripts on your machine:

oloproof scores import --from inspect --answers logs/organism.eval logs/organism.eval
oloproof review latest --criterion deceptive --by NAME

review draws the sample, shows each transcript without the judge's verdict, measures agreement and validates the judge only if it clears the bar you declare. Until it is validated, a policy that requires validated judges, as the one oloproof init writes does, does not decide a rule on it. For a run Oloproof executed, whose cases carry tags in their metadata, oloproof labels disagreements RUN --criterion deceptive --by behavior then shows where the judge and people part, per behavior and by direction: what the judge passed that people failed, and the reverse. Each group's range is its own claim; the report decides nothing.

Write it up

oloproof report latest --paper paper/

This writes figure.svg (each metric's estimate and interval), table.md and table.tex, and methods.md, a paragraph that names each interval method and its confidence, says the intervals are about this finite suite, names missing cases and rare events, says for each judge whether people checked it and how often they agreed, and states what the gate decided. Every number is the stored run's, and the paragraph is built from fixed sentences and recorded facts; no model writes it. In a notebook:

import oloproof

result = oloproof.analyze_inspect("logs/organism.eval", project="analysis")
result  # the figure, the table and the methods paragraph
result.write("paper/")

An auditing agent's transcripts

An auditing tool that runs on Inspect, such as Petri, writes an Inspect log whose scorer gives each transcript a score per dimension, often from 1 to 10. Import it with that range, oloproof scores import --from inspect --scale 1 10 audit.eval, and each dimension becomes a criterion with a bounded mean and its interval. Two such logs, before and after a change to the target, compare on their shared transcripts. The scores stay the judge's word: a judge's scores, as opposed to its pass or fail, cannot be checked against people yet.

What is not built

• Checking a judge's scores against people's, as opposed to its pass or fail verdicts.
• Tags from an Inspect sample's metadata: an imported run's cases carry no tags, so labels disagreements --by groups only runs Oloproof executed.
• A sweep narrower than Bonferroni (Dunnett or closed testing), contrasts between conditions on different suites, and monitor bands narrower in the tail, several monitors combined, or protocols with several thresholds.
• A PNG figure: the figure is SVG only.

What this cannot tell you

Every interval is about the samples in your log. It says what a complete, error-free run of these samples would show, not how the model behaves on other prompts, in deployment, or under an attacker who adapts. A judge checked by people is checked on the cases they labelled and the model that judged them, not on transcripts unlike those. And a rate bounded below 1% on your attack set says nothing about attacks your set does not contain.