Skip to content
Docs
ChangelogPricingOpen console

Guides

From zero to a verdict on your own app

You have a Python function that answers questions, and you want to know if it is good enough to ship. Three commands take you from that function to a release decision, with no YAML written by hand: oloproof init --system writes the project around your function, oloproof check says whether it will work before anything is spent, and oloproof run decides. Everything runs on your machine, with no account and no model calls.

Every output below is from running these commands on the function shown.

1Your function

Any function that takes one case and returns an answer. This one is bot.py, with a helper module it imports:

from prompts import GREETING


def answer(question):
    text = question.lower()
    if "refund" in text:
        reply = "Refunds are available within 30 days of purchase."
    elif "ship" in text:
        reply = "We ship to the US and Canada."
    else:
        reply = "A support agent will follow up within one business day."
    return {"reply": f"{GREETING} {reply}", "topic": "support"}

2Write the project around it

From the directory that holds bot.py:

oloproof init --system bot:answer
Initialized Oloproof project in . around bot:answer
  Each case's input is a string, read from the function's first parameter.

Fill in before a run means anything:
  data/cases.jsonl  3 placeholder cases: give each a real input and, as its expected
                    phrase, words a right answer contains. Add more: see below.
  oloproof.yaml     the check reads the whole answer; if your function returns an
                    object, add `field:` naming the key that holds the text.
  release.yaml      a placeholder bar of 0.80; set the one your release needs.

What this suite can decide, if every case is measured:
  mentions-floor (mentions_expected at least 0.8, 3 cases):
    1 or more pass: INSUFFICIENT_EVIDENCE (not established yet); 0: FAIL
    (ruled out). With 50 cases, PASS from 46; with 100, from 89.

Next: oloproof check, then oloproof run

The reference is imported first, and nothing is written unless it resolves to a function; an error in your module stops here with its file and line. The function is not called. Three files appear: oloproof.yaml (your function and one check), data/cases.jsonl (three placeholder cases) and release.yaml (the bar). The case's input is a string because the first parameter is named question; a parameter named case, inputs, row or annotated as a dict gets an object instead.

The check is contains: a case passes when the answer contains the case's expected phrase. It is a deterministic check, so it needs no measuring against people before it may decide. The suites guide lists every other check and judge.

3Fill in the cases

Replace the placeholders with real questions and the words a right answer contains, one case per line:

{"id": "refund_01", "input": "Can I get a refund?", "expected": {"phrase": "30 days"}}
{"id": "ship_01", "input": "Do you ship to Canada?", "expected": {"phrase": "US and Canada"}}
{"id": "other_01", "input": "My order is late", "expected": {"phrase": "follow up"}}

Three cases cannot pass a bar of 0.80, as the table above says: with three, the best answer is INSUFFICIENT_EVIDENCE. The run below uses 30 cases like these, ten per topic, where 29 passing is enough.

4Check before you run

oloproof check

Before the field was set, it said:

ok   System: callable bot:answer imports.
ok   Called on case refund_01: it answered {"reply": "Thanks for writing in. Refunds are available within 30 days of purchase.", "topic": "support"}

Evaluators on that answer (one case, not a result):
ok   mentions_expected (contains) can read the answer; it passed this case.
note   It reads the whole answer object; `field:` can name one of reply, topic.

So add field: reply under the check in oloproof.yaml, and check again:

ok   System: callable bot:answer imports.
ok   Called on case refund_01: it answered {"reply": "Thanks for writing in. Refunds are available within 30 days of purchase.", "topic": "support"}

Evaluators on that answer (one case, not a result):
ok   mentions_expected (contains) can read the answer; it passed this case.

Code: the system's version hashes bot.py, prompts.py. Edit one and the next run calls the system again.

What this suite can decide, if every case is measured:
  mentions-floor (mentions_expected at least 0.8, 30 cases):
    29 or 30 pass: PASS (established); 19 to 28: INSUFFICIENT_EVIDENCE (not
    established yet); 18 or fewer: FAIL (ruled out). With 50 cases, PASS from
    46; with 100, from 89.

CI: no model key is needed for the evaluators. Keys your own function reads are not checked here.

Ready. Next: oloproof run

What each part means:

  • Imports and answers: The system is built exactly as oloproof run builds it, and called on the first case (--case ID picks another, --no-call skips the call). If it raises, the line names the error and the file and line in your code.
  • Evaluators: Each check that needs no model is run on that one answer. A field it cannot find is named with the fields the answer has. A pass or fail here is one case, not a result. A model judge is never called by check; it says so, and a run calls it.
  • Code: The files whose content is the system's version: its module and the project modules it imports, plus any system.code_paths. A project file loaded while importing or calling the system that is not among them (a module imported by name at run time, say) is listed as not hashed: add it to code_paths, or a change to it reuses stale answers.
  • What it can decide: The same table as oloproof plan. Without a release.yaml, it says a run answers in plain words and decides nothing.
  • CI: Each model the project calls and the secret a CI job needs for it, ANTHROPIC_API_KEY or OPENAI_API_KEY unless api_key_env names another, and none for a server on this machine.

check stores nothing. It exits 2 when something must be fixed and 0 otherwise, and oloproof check --json prints the same findings as one document.

5Run it

oloproof run
Gate: ALLOW (exit 0)
mentions-floor:
  100.0%, range 88.4% to 100.0% (exact binomial interval); 30 of 30 cases

The run's report is longer; these are its verdict and the rule's line. All 30 cases passed, and the range the rate is held to starts at 88.4%, above the bar of 0.80, so the rule says PASS and the gate allows the release. oloproof inspect latest reads it again.

Next

  • Add the gate to pull requests: oloproof init --ci github writes the workflow beside this project (CI guide).
  • Swap the check for a judge when a phrase cannot say what a right answer is (judges guide); oloproof check then names the key CI needs.
  • Already have answers in a file rather than a function? oloproof eval answers.jsonl checks them with no project at all.