チュートリアル：ルーブリックジャッジによるテキスト生成

python3 -m venv .venv
. .venv/bin/activate
pip install oloproof

oloproof init --example generation ticket-summaries
cd ticket-summaries

ticket-summaries/
  app.py                        the summariser under test (baseline)
  app_v2.py                     the candidate change
  judge_server.py               a stand-in judge speaking the OpenAI API on 127.0.0.1
  rubrics/covers_facts.md       the judge's rubric
  oloproof.yaml                 the suite
  release.yaml                  rules for a run
  compare.yaml                  a rule for a comparison
  data/tickets.jsonl            20 cases
  labels/reviewer_verdicts.csv  one person's verdicts on the baseline's summaries
  fill_labels.py                copies those verdicts into a labelling sheet

python judge_server.py --port 8799

stand-in judge on http://127.0.0.1:8799/v1

# app.py
@system(name="ticket-summariser", version="first-sentence")
def summarise(case: dict[str, Any]) -> dict[str, str]:
    return {"summary": sentences(str(case["ticket"]))[0]}

{"id":"t01","input":{"ticket":"Hello. Order 1042 arrived with a cracked screen. I would like a replacement, not a refund."},"expected":{"must_mention":["1042","cracked","replacement"]}}
{"id":"t06","input":{"ticket":"Please cancel my subscription at the end of this month. I am moving abroad."},"expected":{"must_mention":["cancel","end of this month"]}}

version: 1
project: ticket-summaries
dataset: data/tickets.jsonl
system:
  name: ticket-summariser
  version: first-sentence
  callable: app:summarise
  timeout_s: 30
evaluators:
  - type: json_schema
    criterion: format_valid
    field: null
    schema:
      type: object
      required: [summary]
      properties:
        summary: {type: string, minLength: 1}
      additionalProperties: false
  - type: regex
    criterion: short_enough
    field: summary
    pattern: '^.{1,160}$'
    pass_if: match
  - type: rubric_judge
    criterion: covers_facts
    provider: openai_compatible
    model: stand-in-judge
    base_url: http://127.0.0.1:8799/v1
    rubric_file: rubrics/covers_facts.md

PASS when the summary states every fact listed under must_mention in the expected answer, in
words a support agent would recognise, and adds nothing the ticket does not say.
FAIL when any listed fact is missing, changed or contradicted.

version: 1
confidence_level: 0.95
block_on: [FAIL, INSUFFICIENT_EVIDENCE, MANUAL_REVIEW]
require_validated_evaluators: true
rules:
  - id: valid-format
    metric: format_valid
    kind: observed_count
    max_failures: 0
  - id: short-enough
    metric: short_enough
    kind: observed_count
    max_failures: 0
  - id: covers-facts-floor
    metric: covers_facts
    min: 0.60

oloproof run

Run run_01M4... [DECIDED/COMPLETE]
Gate: BLOCK (exit 3)
│ valid-format       │ format_valid │ PASS                  │ observed_failures_within_limit │
│ short-enough       │ short_enough │ PASS                  │ observed_failures_within_limit │
│ covers-facts-floor │ covers_facts │ INSUFFICIENT_EVIDENCE │ evaluator_not_validated        │
covers-facts-floor: the judge (or model or custom evaluator) behind this rule has not been measured against
people yet, so it may not decide.
  Label a sample:  oloproof review run_01M4... --criterion covers_facts --by YOU --sample 20
  Then measure it: oloproof evaluators validate EVALUATOR_ID --by YOU (ids: oloproof evaluators list)
│ format_valid │ 100.0%   │ [83.1%, 100.0%] │ 20 / 20 observed · 0 missing · 0 excluded │
│ short_enough │ 100.0%   │ [83.1%, 100.0%] │ 20 / 20 observed · 0 missing · 0 excluded │
│ covers_facts │ 45.0%    │ [23.0%, 68.5%]  │ 9 / 20 observed · 0 missing · 0 excluded  │
Cache: execution 0 hit/20 miss; judgment 0 hit/60 miss

oloproof inspect RUN_ID --failures

11 of 20 cases failed, errored or did not finish

t01
  output: {"summary": "Hello."}
  covers_facts: failed
    judge text, not verified: missing: 1042, cracked, replacement

t02
  output: {"summary": "I was charged twice for order 2210."}
  covers_facts: failed
    judge text, not verified: missing: 49
...

oloproof labels export RUN_ID --criterion covers_facts --sample 20 --local --out sample.csv

Wrote 20 cases to sample.csv, drawn at random with seed 2701013296, without the judge's verdict.
  This is a local sample, good-faith only, because it was drawn on this machine.
Fill in `passed` (pass or fail) and `labelled_by` on each row you judge, then run `oloproof labels import sample.csv`.

python fill_labels.py sample.csv
oloproof labels import sample.csv

filled 20 rows of sample.csv
Recorded 20 labels from sample.csv (20 measurement).

oloproof evaluators list
oloproof evaluators validate EVALUATOR_ID --by alice

covers_facts  LLM_JUDGE  UNVALIDATED  (declared)  sha256:a662...

covers_facts: sha256:a662... is now VALIDATED
  agreement 95.0% [75.1%, 99.9%] · 19 of 20 labelled cases agreed · 0 labelled but not judged · kappa 0.900
  bias -5.0 points [-32.4, +20.7] · the judge's pass rate minus the people's · 20 cases · 0 labelled but not judged
  passes what people pass 90.0% [55.4%, 99.8%] · the judge passed 9 of 10 cases people passed · 0 labelled but not judged
  fails what people fail 100.0% [69.1%, 100.0%] · the judge failed 10 of 10 cases people failed · 0 labelled but not judged

oloproof gate RUN_ID --policy release.yaml

valid-format: PASS (observed_failures_within_limit)
short-enough: PASS (observed_failures_within_limit)
covers-facts-floor: INSUFFICIENT_EVIDENCE (interval_overlaps_threshold)
  no sample size would make this PASS: the observed rate (0.500) is itself below the threshold (0.600), so more cases would move it toward FAIL
Gate: BLOCK (exit 3)

oloproof run

Gate: ALLOW (exit 0)
│ covers-facts-floor │ covers_facts │ PASS  │ lower_bound_meets_minimum      │
│ covers_facts │ 100.0%   │ [83.1%, 100.0%] │ 20 / 20 observed · 0 missing · 0 excluded │
Cache: execution 0 hit/20 miss; judgment 6 hit/54 miss

version: 1
confidence_level: 0.95
block_on: [FAIL, INSUFFICIENT_EVIDENCE, MANUAL_REVIEW]
require_validated_evaluators: true
rules:
  - id: covers-more-facts
    kind: superiority
    metric: covers_facts

oloproof compare CANDIDATE_RUN_ID BASELINE_RUN_ID --policy compare.yaml

format_valid: +0.0 points [-23.6, +23.6] · 20 paired · 0 missing · 0 excluded
short_enough: +0.0 points [-23.6, +23.6] · 20 paired · 0 missing · 0 excluded
covers_facts: +55.0 points [+13.0, +84.4] · 20 paired · 0 missing · 0 excluded
Decisions
  covers-more-facts  covers_facts  superiority  PASS  difference_above_zero
Gate: ALLOW (exit 0)

# live_judge.yaml
type: rubric_judge
criterion: covers_facts
provider: openai_compatible
model: llama3.1
base_url: http://localhost:11434/v1
rubric_file: rubrics/covers_facts.md

oloproof evaluators try live_judge.yaml

covers_facts: draft sha256:b88a... on 20 labelled cases · 20 judged now, 0 from cache, 11 errored
  agreement 88.9% [19.1%, 99.9%] · 8 of 9 labelled cases agreed · 11 labelled but not judged · kappa 0.769