チュートリアル: 模擬された顧客を相手にするサポートデスク

python3 -m venv .venv
. .venv/bin/activate
pip install oloproof

oloproof init --example simulated-users support-desk
cd support-desk

support-desk/
  desk.py                      the support desk under test (baseline)
  desk_v2.py                   the candidate change
  user_server.py               a stand-in simulated user speaking the OpenAI API on 127.0.0.1
  oloproof.yaml                the suite: the simulated user and four evaluators
  release.yaml                 rules for a run, with the realism bar
  compare.yaml                 a rule for a comparison
  data/customers.jsonl         30 personas
  labels/realism_verdicts.csv  one person's realism verdicts on the baseline's conversations
  fill_realism.py              copies those verdicts into a labelling sheet
  logged.py                    replays logged chats with human agents
  real.yaml                    a suite of 20 of those chats
  logged_release.yaml          a rule for the logged chats
  data/logged_chats.jsonl      the 20 chats

# desk.py (excerpt)
def talk(case: dict[str, Any]) -> dict[str, Any]:
    user = current_case().simulated_user()
    desk = Desk()
    message = user.opening()
    while message is not None:
        message = user.reply(desk.answer(message))
    return {"need": desk.need, "order": desk.order}

{"id": "c25", "input": {"persona": {"description": "A retired teacher who ordered a cookbook last week. Writes short, polite messages.", "goal": "Have the order of a cookbook delivered to my new address.", "facts": {"order": "B-35", "postcode": "10275"}}}}

version: 1
project: support-desk
dataset: data/customers.jsonl
system:
  name: support-desk
  version: baseline
  callable: desk:talk
  # A model plays each case's persona; here the stand-in in user_server.py.
  simulated_user:
    provider: openai_compatible
    model: stand-in-user
    base_url: http://127.0.0.1:8797/v1
    max_turns: 6
evaluators:
  # The desk answered every message the customer sent.
  - type: conversation_completed
  # The customer said they got what they came for.
  - type: goal_reached
  # ... within three of their own messages.
  - type: turns_to_goal
    max_turns: 3
  # No answer promises what the shop cannot promise.
  - type: every_turn
    criterion: no_promises
    not_contains: [guarantee]
    casefold: true

python user_server.py --port 8797

version: 1
confidence_level: 0.95
block_on: [FAIL, INSUFFICIENT_EVIDENCE, MANUAL_REVIEW]
require_validated_evaluators: true
minimum_simulated_user_realism: 0.80
rules:
  - id: no-promises
    metric: no_promises
    kind: observed_count
    max_failures: 0
  - id: resolved
    metric: goal_reached
    min: 0.80

oloproof run

Gate: BLOCK (exit 3)
│ no-promises │ no_promises  │ INSUFFICIENT_EVIDENCE │ simulated_user_not_validated │
│ resolved    │ goal_reached │ INSUFFICIENT_EVIDENCE │ simulated_user_not_validated │
│ conversation_completed │ 100.0%   │ [88.4%, 100.0%] │ 30 / 30 observed · 0 missing · 0 excluded │
│ goal_reached           │ 73.3%    │ [54.1%, 87.8%]  │ 22 / 30 observed · 0 missing · 0 excluded │
│ goal_within_3_turns    │ 73.3%    │ [54.1%, 87.8%]  │ 22 / 30 observed · 0 missing · 0 excluded │
│ no_promises            │ 60.0%    │ [40.6%, 77.4%]  │ 18 / 30 observed · 0 missing · 0 excluded │

oloproof simulated-users RUN_ID

  Realism: no conversation has been drawn and labelled yet.
  Not validated: nothing labelled yet, against a bar of 80.0%.
Features (median [quartiles]; descriptive, never a gate):
  user messages           2 [1, 2]
  user message length     49 [10, 67]
  How the simulated conversations ended: goal_reached 73%, user_left 27%

oloproof labels export RUN_ID --criterion simulated_user_realistic --sample 30 --local --out realism.csv
python fill_realism.py realism.csv
oloproof labels import realism.csv
oloproof simulated-users RUN_ID

  Realism: 29 of 30 drawn conversations labelled realistic (30 labelled), [82.8%, 99.9%] at 95%
  Validated: the lower bound clears the bar of 80.0%.

oloproof gate RUN_ID --policy release.yaml

no-promises: FAIL (observed_failures_exceed_limit)
resolved: INSUFFICIENT_EVIDENCE (interval_overlaps_threshold)
Gate: BLOCK (exit 1)

oloproof run

Gate: ALLOW (exit 0)
│ no-promises │ no_promises  │ PASS  │ observed_failures_within_limit │
│ resolved    │ goal_reached │ PASS  │ lower_bound_meets_minimum      │

oloproof compare CANDIDATE_RUN_ID BASELINE_RUN_ID --policy compare.yaml

goal_reached: +26.7 points [+1.2, +54.1] · 30 paired · 0 missing · 0 excluded
  resolves-more  goal_reached  superiority  PASS  difference_above_zero

oloproof run --config real.yaml --policy logged_release.yaml
oloproof simulated-users BASELINE_RUN_ID --real REAL_RUN_ID

                          simulated (run_01M52Q1C)    real (run_01M52Q25)
  user messages           2 [1, 2]                    3 [3, 4]
  user message length     49 [10, 67]                 13 [4, 32]

  simulated_user:
    provider: openai_compatible
    model: qwen2.5vl
    base_url: http://localhost:11434/v1
    max_turns: 6