Fairness across groups

A model can clear its accuracy bar for every group and still be harsher on one of them: twice the false negatives, or half the selections. oloproof fairness reads a file of decided cases and asks how far apart the groups are. It answers with the same four states as every other rule. The parity method was admitted by an independent audit on 3 November 2026, on the terms this page states: cases independent, groups declared first, each rule about the cases where its attributes are recorded, and parity measured against your labels.

The file and the declaration

One CSV or JSONL file with a row per case: an id, the label (0 or 1), the prediction (0 or 1, empty when the system gave none), optionally a probability, and one column per protected attribute.

fairness:
  name: loan-screen
  cases: decided.csv
  label: label
  prediction: prediction
  probability: score
  attributes:
    sex: [female, male]
    age_band: [under_30, 30_to_60, over_60]
  intersections:
    - [sex, age_band]
  min_cell: 30
  rules:
    - {id: fnr-by-sex, rate: false_negative, attribute: sex, max_group_gap: 0.05}
    - {id: fnr-crossed, rate: false_negative, attribute: [sex, age_band], max_group_gap: 0.10}

oloproof fairness fairness.yaml --policy release.yaml --json

The interval assumes each case is a different person. Name the column that identifies the person with unit: person_id and a file where one person has several cases is refused; without it, the answer states the assumption.

The rate is one of error, false_positive (over cases labelled 0), false_negative (over cases labelled 1) or selection (predicted 1, over every case). Name the groups before you read the file. Each rule reads only the cases whose own attributes are recorded with a declared value; a case missing an attribute is left out of the rules that use it and kept by the others, and each answer counts what it left out and why. When an attribute is missing more often for some people than others, the answer is about the cases where it is recorded, which is a different population: read the count before the verdict. A rule that leaves out more than max_left_out of the file (10% by default; set it on the rule) reads INSUFFICIENT_EVIDENCE, because the recorded cases may no longer stand for the rest. The exit code is the worst rule's: 0 PASS, 1 FAIL, 3 INSUFFICIENT_EVIDENCE, 4 MANUAL_REVIEW.

How it decides

Each group's rate gets the exact interval Oloproof uses for one rate, at the error budget divided by the number of declared groups. Every difference between two groups is bounded from those, all at once, so no pair needs its own correction. A max_group_gap rule passes when every gap's upper end is at or below the bar, and fails when some gap is above the bar with its whole interval. A missing prediction stays in its group and is counted at its worst.

Small groups and crossed groups

Crossing two attributes multiplies the cells, and some will be small. A cell with fewer cases with a prediction than min_cell (cases whose prediction is missing do not count toward it) is reported INSUFFICIENT_EVIDENCE on its own line, still counts in the split, and keeps the rule from passing, because the largest gap could be there. Collect more cases for that cell, or declare a coarser crossing before the run.

Floors and gaps

Use worst_group for a floor every group must clear ("each region resolves at least 80%"), and max_group_gap for a ceiling on the distance between groups ("false negatives differ by at most five points"). A governance requirement for unbiased outcomes is evidenced in part by the gap, for the rate you chose; a requirement that every group be served well, by the floor. Neither alone shows that a system is fair. Most systems need both.

Calibration by group

With a probability column, each group's expected calibration error is reported for every attribute and declared crossing, over calibration_bins equal-width bins of the probability (10 by default). It never gates: no admitted method bounds it, and in a small group it is noisy.

What this cannot tell you

• Which rate is the fair one to equalize: that is your decision, and worth recording with the attributes you chose.
• Anything about groups you did not declare, or about people counted twice (one person, several cases).
• Whether calibration differs by more than chance between groups.
• Whether the labels themselves are biased: parity is measured against the labels you bring.