Online experiments (A/B tests)

Your offline comparison said the new version is better. An A/B test in production asks the same question of real users. Your own router randomizes units (users or sessions) between two arms; oloproof experiment reads its logs and answers with the same four states as every other rule. The method is admitted by an independent audit on stated terms: observed_until is your word for when the outcome log was extracted, units were randomized independently, no unit's outcome depends on another's arm, outcomes did not drift while the split changed, and each look reads the whole log. Outside them the answer is not covered.

The logs and the declaration

Two files, CSV or JSONL: an assignment log with one row per unit (the unit column you declare, and arm), and an outcome log with the unit and its value.

experiment:
  name: checkout-v2
  unit: user
  allocation: {control: 0.5, treatment: 0.5}
  control: control
  treatment: treatment
  assignments: logs/assignments.csv
  outcomes: logs/outcomes.csv
  kind: rate
  rules:
    - {id: lift, kind: superiority}
    - {id: no-harm, kind: non_inferiority, margin: 0.01}

oloproof experiment experiment.yaml --policy release.yaml --json

kind: mean takes a declared range: [low, high]; direction: max reads lower as better. An assigned unit with no outcome row is missing and bounded at its worst on each side, unless missing_outcome: zero says no row means zero. Log a row for every unit, zeros included: a conversion log that holds only converters reads every other unit as missing, and the answer stays INSUFFICIENT_EVIDENCE. Zero-fill assumes every counted unit's outcome window has closed; a unit assigned an hour ago has not converted yet, and counting it as zero favours whichever arm converts faster. Declare outcome_window_hours, as_of and observed_until (when the outcome log was extracted), with an assigned_at column in the assignment log, and only units whose window closed by as_of are counted; missing_outcome: zero is refused without them, whatever the monitoring. Write every time with an offset or Z (2026-10-11T00:00:00Z); a time without one is refused. as_of may not be later than observed_until, and neither may be in the future: an outcome after the extraction is not in the log. The window chooses the question: a 48-hour window answers "converted within 48 hours", so it must cover the delay of the outcome you mean. Unit ids are compared as text after trimming spaces, with 42.0 read as 42. The exit code is the worst rule's: 0 PASS, 1 FAIL, 3 INSUFFICIENT_EVIDENCE, 4 MANUAL_REVIEW.

What it checks first

• Sample ratio. The arms' unit counts are tested against the declared allocation (an exact binomial test for two arms, chi-square for more, threshold 0.001). A mismatch makes every rule MANUAL_REVIEW: a broken split means the arms are no longer comparable, and more data of the same kind does not repair it. Find the cause (a redirect that drops users, a bot filter applied to one arm) before reading any difference.
• A unit in two arms. Assignment leaked; every rule reads MANUAL_REVIEW.
• Several outcome rows for one unit. Refused, because events of one user are not independent. Declare aggregate: unit_mean to make each unit's own mean its outcome: the answer is then about the average user, not the average event.
• An empty arm. INSUFFICIENT_EVIDENCE until it has outcomes.
• Outcome rows that match no assigned unit, under missing_outcome: zero. The zeros rest on the join, so a broken join would read real units as zeros; every rule reads MANUAL_REVIEW until the ids agree. Under the default they are counted and left out.

How it decides

Each arm's mean gets the interval Oloproof already uses for one run (exact Clopper-Pearson for a rate, the betting bounded mean otherwise), each at half the error budget, and the difference is bounded by the far ends of the two. It holds at every sample size and is wider than a textbook normal interval. A superiority rule passes when the whole interval is above zero; a non-inferiority rule passes when its lower end is at or above minus the margin.

monitoring: anytime uses the betting interval for both arms, which holds however often you look and whenever you stop, as long as each look passes the whole log up to that moment: never drop or filter rows by period after reading them. The default, fixed, is read once at the sample size you planned.

Several treatments

Declare treatments: [b, c] instead of treatment: and give each rule the treatment it compares (treatment: b). Each arm's interval is taken at 1 - alpha/m for the m arms, so every treatment minus the control is covered at once by the union bound: one rule per treatment, one claim each, and the family holds together. Each interval is wider than with two arms, because each arm pays for the others. With several treatments, the report names the best among those that pass every rule naming them, and whether it is separated from every other passing treatment: each difference, bounded by the same simultaneous intervals, excludes zero. Ties keep the declared order and are never separated. The sample ratio check already reads every arm. The joint guarantee covers the arms of one experiment file: splitting the treatments across several files gives it up.

Ratio metrics

A click-through rate is clicks over sessions, and a user with many sessions is not many independent users. Declare the ratio over one row per unit, with ranges that every unit's values lie in:

  ratio: {numerator: clicks, denominator: sessions, numerator_range: [0, 50], denominator_range: [1, 50]}
  aggregate: unit_sum   # only if a unit has several rows: they are added

The estimand is each arm's mean numerator over its mean denominator. Both means are bounded by the admitted interval, each at its share of the error budget, and the ratio by dividing their ends, so it holds at every sample size with no normal approximation. A denominator whose interval reaches zero gives no interval. A value outside its range is a configuration error naming the unit, never clipped. The delta method's interval, usually narrower, is shown beside it and never decides.

A covariate from before the test

An outcome that tracks something measured before the test (last month's spend, last week's sessions) varies less once that is taken out. Bring each unit's value from before the test and a coefficient fixed before this test's outcomes are read:

  covariate: {values: logs/pre_period.csv, range: [0, 500], theta_from: logs/history.csv}

The decided outcome is Y - theta (X - c). theta is declared (theta:, with an optional center:, the middle of the range by default), or fitted on a history file from a period before the test (covariate and outcome columns), never on this test's own outcomes. Because X was measured before assignment, and which units have an X was fixed before assignment too, the adjusted difference has the same estimand. A unit with no value in the file is not adjusted. A valid adjustment can still be wider than none, because the adjusted outcome's range is wider, so the report shows both intervals and says which is narrower; it claims no saving.

Novelty and interference, shown

novelty: {exposure_column: exposed_at, time_column: at, days: 7} reads each unit's first exposure from the assignment log and each outcome row's time, and shows the difference in the first days of exposure and later. A decided early difference with an undecided or opposite later one is named "possibly a novelty effect". The split never decides a rule: units without rows in a band are left out of it, not bounded at their worst. It takes one outcome per row, not a ratio or a covariate.

cluster_column: team counts the teams that hold units of several arms and asks whether one unit's outcome can depend on another's arm. If it can, randomize by the team and declare it as the unit.

Several treatments, ratios and covariates were admitted by an independent audit and its confirmation on 3 November 2026, on the same terms as two arms.

Offline and online

Name the offline comparison and its metric (offline_comparison: <id>, offline_metric: pass_rate) and the answer reports whether the two decided directions agree. It is reported and never decides anything: a disagreement is a reason to look at the suite, not a verdict on the release.

Pitfalls

• Peeking. Reading a fixed-horizon test every day and stopping on a good one inflates false wins. Plan the sample size, or declare monitoring: anytime.
• Novelty and learning. Users react to anything new, then settle. A week-one lift can fade; run long enough to cover the cycles your users have, and declare novelty: to see the split.
• Interference. The interval assumes one unit's outcome does not depend on another's arm. Shared inventory, marketplaces, social features or a cache warmed by one arm break it; randomize by a larger unit (a region, a team) when they do, and declare cluster_column: to see how mixed the clusters are.
• Drift under a changing split. The interval assumes the allocation stayed constant over the test, or that outcomes did not drift while it changed. A ramp from 10% to 50%, or 90/10 then 10/90 over a weekday and a weekend, weighs the periods differently in each arm and mixes a time effect into the difference; the sample ratio check reads pooled counts and does not see it. Keep the split fixed, or decide only on the period after the ramp.
• Changing the unit. Randomize and analyse by the same unit. Randomizing users and counting sessions as independent narrows the interval falsely.

What this cannot tell you

• Whether the result holds for users outside the test's window, region or traffic.
• Anything about a metric you did not declare before the test; a metric chosen after the fact is a description.
• Whether your router really randomized: the sample ratio check catches large faults, not subtle ones.
• Which treatment is best among those that did not pass: only passing treatments are ranked. Stratified allocation and switchback designs are not built.