ゼロからご自身のアプリの判定まで

from prompts import GREETING


def answer(question):
    text = question.lower()
    if "refund" in text:
        reply = "Refunds are available within 30 days of purchase."
    elif "ship" in text:
        reply = "We ship to the US and Canada."
    else:
        reply = "A support agent will follow up within one business day."
    return {"reply": f"{GREETING} {reply}", "topic": "support"}

oloproof init --system bot:answer

Initialized Oloproof project in . around bot:answer
  Each case's input is a string, read from the function's first parameter.

Fill in before a run means anything:
  data/cases.jsonl  3 placeholder cases: give each a real input and, as its expected
                    phrase, words a right answer contains. Add more: see below.
  oloproof.yaml     the check reads the whole answer; if your function returns an
                    object, add `field:` naming the key that holds the text.
  release.yaml      a placeholder bar of 0.80; set the one your release needs.

What this suite can decide, if every case is measured:
  mentions-floor (mentions_expected at least 0.8, 3 cases):
    1 or more pass: INSUFFICIENT_EVIDENCE (not established yet); 0: FAIL
    (ruled out). With 50 cases, PASS from 46; with 100, from 89.

Next: oloproof check, then oloproof run

{"id": "refund_01", "input": "Can I get a refund?", "expected": {"phrase": "30 days"}}
{"id": "ship_01", "input": "Do you ship to Canada?", "expected": {"phrase": "US and Canada"}}
{"id": "other_01", "input": "My order is late", "expected": {"phrase": "follow up"}}

oloproof check

ok   System: callable bot:answer imports.
ok   Called on case refund_01: it answered {"reply": "Thanks for writing in. Refunds are available within 30 days of purchase.", "topic": "support"}

Evaluators on that answer (one case, not a result):
ok   mentions_expected (contains) can read the answer; it passed this case.
note   It reads the whole answer object; `field:` can name one of reply, topic.

ok   System: callable bot:answer imports.
ok   Called on case refund_01: it answered {"reply": "Thanks for writing in. Refunds are available within 30 days of purchase.", "topic": "support"}

Evaluators on that answer (one case, not a result):
ok   mentions_expected (contains) can read the answer; it passed this case.

Code: the system's version hashes bot.py, prompts.py. Edit one and the next run calls the system again.

What this suite can decide, if every case is measured:
  mentions-floor (mentions_expected at least 0.8, 30 cases):
    29 or 30 pass: PASS (established); 19 to 28: INSUFFICIENT_EVIDENCE (not
    established yet); 18 or fewer: FAIL (ruled out). With 50 cases, PASS from
    46; with 100, from 89.

CI: no model key is needed for the evaluators. Keys your own function reads are not checked here.

Ready. Next: oloproof run

oloproof run

Gate: ALLOW (exit 0)
mentions-floor:
  100.0%, range 88.4% to 100.0% (exact binomial interval); 30 of 30 cases