Skip to content
Docs
ChangelogPricingOpen console

Guides

What an evaluation cannot tell you

Every Oloproof result is a measured claim with stated limits. This page lists the limits that hold for every kind of system; each guide page by system adds its own under "What this cannot tell you".

What no result tells you

  • Inputs unlike the suite. A decision is a claim about the population the suite samples, and it is only as good as the suite is a fair sample of what your system will meet. Oloproof cannot check that; a new kind of input or next month's traffic lies outside it (How far a result reaches).
  • That a check can never fail. A check that passed on all 300 cases bounds how often it fails (at most about 1.2% at 95% confidence); it is not a proof.
  • How your system answers next time. A run measures one answer per case unless you ask for replicates: (Writing a suite).
  • What an unchecked judge's scores mean. A judge never compared with people's labels stays UNVALIDATED, and its pass rate is the judge's opinion. A policy with require_validated_evaluators: true refuses to let it decide (Judges).
  • That a rule failed when it says "not established yet". INSUFFICIENT_EVIDENCE means the interval still straddles the bar, not that the system missed it; oloproof plan says how many more cases would decide it.
  • Cost or latency, unless a rule asks. A release decision covers the rules your policy declares; a budget on a latency quantile or on cost per accepted task is a max: rule you add (Gating).
  • Whether a reused answer is still current. A cached output or judgment is the earlier one, reused because its inputs hash the same; a change the hash cannot see (a model changed behind the same name, a file the system reads but does not import, an environment variable) is not seen (Versions, cache keys and invalidation).
  • A slice you chose after looking. Slice results are claims under the declared correction for the slices you declared; a cut picked because it looked interesting carries no such guarantee, and slices never gate (Slices).
  • Why a case failed. Evaluators say what happened; a cause takes an intervention beside a control, and even then Oloproof names a factor to test, not an established cause.
  • Cases ordered in time as if they were independent. A suite that declares time_order reads MANUAL_REVIEW rather than an interval that assumes independent cases (Clustered cases).
  • A metric with no admitted interval. A rule that would need one returns MANUAL_REVIEW or INSUFFICIENT_EVIDENCE with its reason rather than borrowing an unvalidated method.

By kind of system

Where to go next