Risk ceilings

A ceiling bounds how often, or how much, something goes wrong: unsafe answers, leaked data, attacks that succeed, a cost or a latency. Oloproof measures the ceilings your policy declares, on the cases your suite holds. It does not write attacks or search for new failures; if another tool does, Oloproof reads that tool's per-case results and decides the ceiling on them.

Two kinds of ceiling

version: 1
rules:
  # A rate: the share of answers a check flags as unsafe, at most 2%.
  - id: unsafe-ceiling
    metric: unsafe
    max: 0.02
  # A count: no case in this suite may leak a customer's data.
  - id: no-leak
    metric: no_leak
    max_failures: 0

max: decides an interval. The rule passes when the interval's upper bound is at or below the ceiling, fails when its lower bound is above it, and is INSUFFICIENT_EVIDENCE in between. The interval is two-sided at the policy's confidence_level, 95% by default, and a ceiling reads its upper end alone, so a system whose true rate is above the ceiling passes at most 2.5% of the time, when its cases are independent (a suite that declares clusters is read with an interval that allows for them, Clusters). For a pass or fail criterion, the rate is the share of cases that pass, so the criterion must pass when the thing to bound happened (unsafe passes on an unsafe answer). A floor on the opposite criterion decides the same: min: 0.98 on a criterion that passes on a safe answer is the same rule as max: 0.02 on one that passes on an unsafe one.

max_failures: counts. It reads the cases that did not pass in the suite as it ran, with no interval, and claims nothing beyond those cases. It fails as soon as the count is over the limit; a case with no verdict counts against it until it has one, so a suite with missing verdicts that could take it over the limit is INSUFFICIENT_EVIDENCE, not a pass. Use it for what must never happen in a suite you wrote on purpose, where a rate over a population would mean nothing.

How many cases a ceiling needs

A low ceiling needs many cases, even when nothing goes wrong. At 95%, with the exact interval Oloproof uses for a pass or fail rate:

With no failure seen, it takes about 3.7 divided by the ceiling. A suite smaller than that cannot pass the rule whatever the system does, and the gate says so: an undecided ceiling is INSUFFICIENT_EVIDENCE, with a line saying how many more cases would decide it.

Reading a red-team tool's results

A red-team tool writes the attacks and grades each response. Its per-case grades are what a ceiling needs, and oloproof scores import reads them (A second opinion). The example's recipes/ folder holds one for promptfoo:

oloproof init --example second-opinion second-opinion
cd second-opinion
promptfoo redteam generate
promptfoo eval -c redteam.yaml --output before.jsonl
python recipes/promptfoo_redteam.py before.jsonl before.csv

Each attack becomes one case, and its grade becomes resisted, which passes when the target resisted. Rows promptfoo could not grade carry no verdict and count as missing. A policy for it:

version: 1
rules:
  - id: attacks-resisted
    metric: resisted
    min: 0.98
  - id: no-successful-attack
    metric: resisted
    max_failures: 0

oloproof scores import before.csv --policy release.yaml

promptfoo grades with a model, and Oloproof treats it as it treats any judge: until people have labelled a sample of its grades, it cannot decide a release, and the rules stop at evaluator_not_validated (Checking the judge). To compare two versions, run the same redteam.yaml on both: a case is the attack's place in that file, and generating the attacks again makes other cases.

What a ceiling does not say

• A rate ceiling is about the population your suite samples. A red-team suite is attacks someone wrote, not a sample of real traffic, so its rate is a rate over those attacks; the count is often the clearer rule there.
• Neither kind covers a failure no case exercises. The bound is on what was run.
• A passing ceiling is evidence for your release decision, not a certification of safety.

Where to go next

• Gating a release: the policy file, block and warn, and exit codes.
• A second opinion: the scores file every recipe writes.
• Judges: measuring a grader against people before it may decide.