Validating a trained model

A trained model can score well on a suite it has already seen, flip its answer when an input moves a little, and get a lucky seed. Oloproof checks each of these before the model ships: the suite against the training data, each prediction under declared noise, and the training recipe over several seeds. Copy the example with oloproof init --example model-validation: a loan model trained by NumPy, offline, on synthetic data.

The suite against the training data

oloproof leakage check --training data/training.jsonl --record

Each case is compared with every training row by exact match, after lower-casing strings and collapsing whitespace (--round rounds numbers, --field names the keys to compare). The training file is read on your machine and never stored or sent; the record keeps case ids and counts. Declare leakage: required in release.yaml and the gate is blocked (exit 3) until a clean check of this suite is recorded. A case found in the training data makes the evidence invalid; it is not a FAIL of the model. Near-duplicates are not checked.

Predictions under noise

metamorphic:
  transforms:
    - name: noise
      field: features
      perturb: {kind: numeric_noise, scale: {income: 1.0, debt_ratio: 0.02}, seed: 7}
evaluators:
  - {type: metamorphic_relation, relation: same_label, transform: noise, field: label, passes_when: flipped}

Each case gains a copy with seeded noise on the named numbers, and flips_under_noise is the share of cases whose label changed. categorical_swap replaces values by a declared map, and blur, brightness, occlusion and jpeg change images. A regressor uses same_score_within with a tolerance. People cannot judge whether noise on a row keeps its meaning, so a numeric perturbation is declared: its rate is reported, and a rule on it reads INSUFFICIENT_EVIDENCE with transform_declared. An image transform gates once people's labels show it keeps each image's meaning, as a text transform does. The example's image/ project shows both: a blur its labels check, whose flip rate fails a ceiling, and a grey patch its labels show sometimes hides the scratch, whose rule withholds.

The recipe over seeds

The checkpoint you ship is decided on its own run, as any system is. A second question is whether the way you train is good enough: retrain next month and you get another draw. Train k checkpoints with k seeds, run each on the same suite, and name the runs:

seeds:
  recipe: loan-model
  checkpoints: {seed_1: latest~4, seed_2: latest~3, seed_3: latest~2, seed_4: latest~1, seed_5: latest}
  rule: {id: recipe_accurate, criterion: correct, min: 0.8}

oloproof seeds seeds.yaml --policy recipe.yaml reads the expected pass rate over seeds, with an interval that counts both the cases the suite holds and the seeds you tried. In the example five seeds score 85.7% to 90.3%, and the recipe reads 88.2%, 83.2% to 91.9%: PASS against 80%. The interval is approximate, so the policy needs allow_approximate_methods: true, and otherwise the answer is MANUAL_REVIEW. The policy must hold at least one rule; the command reads its confidence level, the opt-in and its judge bars, and nothing else. It also lists the cases whose verdict changes from one seed to another, and --html FILE writes one offline page with every case under every checkpoint.

Every checkpoint counts. A seed whose training failed is kept, and each case it did not answer counts as a fail for the low end and a pass for the high end, so it widens the interval. Fewer than three seeds read INSUFFICIENT_EVIDENCE; about five are needed to decide anything. A judge the gate would not trust does not decide a recipe either. Every checkpoint must run the same suite, with one system version each, and a suite that declares groups is refused.

What this cannot tell you

• Perturbations nobody declared. Only the declared transforms are tried, at the declared sizes, and a low flip rate says nothing about larger changes or an attacker's search.
• A proven robustness bound. No bound is proved for any input; the flip rate is a rate over the suite's cases.
• How the model was trained. Oloproof cannot see whether the seeds were chosen before any checkpoint was evaluated, whether failed runs were thrown away, or whether the checkpoints differ only by their seed.
• A failure rarer than about one seed in k. If the recipe fails to train one time in ten and none of five seeds failed, the interval cannot know.
• Leakage the training file does not show: near-duplicates, a resized image, or rows used for tuning but left out of the file.
• Leakage in the seed runs. oloproof seeds does not read leakage: required; the checkpoint you ship is gated on its own run, which does.

What every evaluation cannot tell you is on its own page.