People with and without the system

An AI system that advises people is good only if the people decide better with it. A system that is right 90% of the time can still make its users worse, if they follow it where it is wrong and stop thinking where they used to. oloproof human-ai report answers two questions from a file of people's decisions: do people decide better with the system's output shown than without it, and how do they rely on it?

The study

Give the same cases to several people. Each person decides some cases with the system's output shown and the others without, the split chosen at random per person, in a random order. The split must be random, or the whole suite in each condition: a split chosen by the person, by the system or by difficulty makes the answer wrong in a way the interval does not cover, and Oloproof cannot see it in the file. Each person decides alone; a team that discusses cases counts as one person. Every case has a reference answer, decided independently. Two designs are decided, and they answer different questions. Crossed: each person sees each case once, in one condition, so the answer is what the output does to a decision. Before and after: each person decides a case alone first and then again after seeing the output, so the answer is a second look with the output against a first look without it, which mixes the output's effect with the effect of looking again. In that design the decision without the output must come first; a file that shows otherwise is named. A file where each case was decided once in each condition by two different people is named sparse, and is described but never decided.

Write one row per decision, as CSV with a header or as JSONL:

Decisions, references and outputs are compared as text after trimming spaces, so write each label one way. Copy the example with oloproof init --example human-ai: thirty claims triaged by eight people, drawn from a seeded generator.

oloproof human-ai check decisions.csv
oloproof human-ai report decisions.csv
oloproof human-ai gate decisions.csv --policy policy.yaml

check reads the file and says what it can answer. report gives every measure and decides nothing. gate decides each rule of a policy and exits 0 on PASS, 1 on FAIL, 3 on INSUFFICIENT_EVIDENCE and 4 on MANUAL_REVIEW. Each takes --json. The file is read on your machine and never stored or sent.

With minus without

The main answer is the average, over people, of each person's accuracy with the output shown minus their own accuracy without it. Each person counts once, however many cases they decided, so a few people who do most of the work cannot hide what the output does to everyone else. The answer is about the people who will use the system, if the people in the study were drawn from them, on the cases in the study, and on others only as far as they are like them.

The interval is the one Oloproof uses for every bounded average, applied to the people's gains. It makes no assumption about how gains are spread: if one person in five follows the system into errors, the interval allows for it. Each side is right with the stated confidence at every number of people, so a minimum bar is not passed by luck more often than it says. The price is width: with six people the interval is about 1.2 wide on a scale from -1 to 1, with ten people about 0.8, and below about forty people a rule will rarely decide unless the gain is large. When the interval is wider than 0.5 the report says the study cannot decide a useful bar and roughly how many people it would need. An interval that treated every decision as independent would be far too narrow (the coverage grid measured it covering the truth as rarely as 39% of the time, against a stated 95%). A missing decision is filled as wrong with the output for the lower bound and as right with it for the upper bound, so the interval holds whatever the missing decisions were, and the report shows the estimate's range over both fills.

The method was admitted by an independent audit on 3 November 2026, on the terms above: people who decide alone, a random split, and in a before-and-after study the decision without the output first. A rule on accuracy_difference decides PASS, FAIL or INSUFFICIENT_EVIDENCE. It needs at least two people who decided in both conditions; with fewer, a rule reads INSUFFICIENT_EVIDENCE.

Reliance

Every rate counts decisions; its denominator says which.

Each rate has the exact interval Oloproof uses for every rate, with missing decisions filled both ways. That interval assumes independent decisions. When a person or a case appears more than once in a rate's denominator, as in any real study, the rate is reported and a rule on it reads MANUAL_REVIEW: repeated people make the interval too narrow, and Oloproof will not decide on it.

Time to decision is the median number of seconds per condition, with its interval, reported and never gated.

A policy

confidence: 0.95
rules:
  - id: people-decide-better
    metric: accuracy_difference
    minimum: 0.0
  - id: few-follow-a-wrong-suggestion
    metric: over_reliance
    maximum: 0.5

Each rule names one metric and one bar. A minimum passes when the interval's lower bound is at or above it and fails when the upper bound is below it; a maximum is the mirror. Anything else is INSUFFICIENT_EVIDENCE.

What this cannot tell you

• It speaks for the people the study's people were drawn from, on cases like the study's. People trained differently, or chosen because they like the system, are another population.
• A before-and-after study measures a second look with the output against a first look without it, not the output alone.
• People who know they are being studied may check the system more carefully than they do at work.
• The reference answers are taken as right. If they come from the same people, or from the system, the study measures agreement, not accuracy.
• Reading decisions from the hosted review queue is not built: export them to the file above.