Guides

Tutorial: a support desk against simulated customers

Evaluate a conversational assistant against users who react to what it says. Thirty simulated customers, each with a goal and the facts they know, talk with a shop's support desk; Oloproof measures whether each got what they came for, withholds every decision until a person has found the simulated customers realistic, and then decides a release and a comparison. Every step runs on this machine with no model and no network, and an optional step swaps in a real model as the customers.

What you will build

A shop's support desk answers refunds, delivery questions and address changes. Three questions decide whether a version may ship:

Did each customer get what they came for? goal_reached, from a check after each answer.
Did any answer promise what the shop cannot promise? every_turn reads every answer for the word "guarantee".
Can the answers be trusted to say anything about real customers? Only once a person has read a sample of the conversations and found the simulated customers realistic.

Terms such as case, run, gate and suite are defined in Concepts; everything about simulated users is in Simulated users.

Prerequisites

Python 3.11 or later, and Oloproof:

python3 -m venv .venv
. .venv/bin/activate
pip install oloproof

The example project, which ships with the package. Copy it into a new directory and work there:

oloproof init --example simulated-users support-desk
cd support-desk

Port 8797 free for the stand-in simulated user (change it in both places if not).

Every step until "Optional: a real model as the customers" is offline and deterministic: no API key, no provider account, no cost.

The files

support-desk/
  desk.py                      the support desk under test (baseline)
  desk_v2.py                   the candidate change
  user_server.py               a stand-in simulated user speaking the OpenAI API on 127.0.0.1
  oloproof.yaml                the suite: the simulated user and four evaluators
  release.yaml                 rules for a run, with the realism bar
  compare.yaml                 a rule for a comparison
  data/customers.jsonl         30 personas
  labels/realism_verdicts.csv  one person's realism verdicts on the baseline's conversations
  fill_realism.py              copies those verdicts into a labelling sheet
  logged.py                    replays logged chats with human agents
  real.yaml                    a suite of 20 of those chats
  logged_release.yaml          a rule for the logged chats
  data/logged_chats.jsonl      the 20 chats

Run every command from support-desk/.

The desk keeps its own loop

# desk.py (excerpt)
def talk(case: dict[str, Any]) -> dict[str, Any]:
    user = current_case().simulated_user()
    desk = Desk()
    message = user.opening()
    while message is not None:
        message = user.reply(desk.answer(message))
    return {"need": desk.need, "order": desk.order}

The desk asks for the customer's first message, answers it, and hands each answer back for the next message, until the customer stops. It never sees the persona. Oloproof records what passed between them; the desk records nothing itself. This baseline cannot change a delivery address, and it promises refunds "guaranteed within 5 days".

The personas

Each case is one customer: 12 want a refund, 10 want to know when an order arrives, and 8 have moved. For example:

{"id": "c25", "input": {"persona": {"description": "A retired teacher who ordered a cookbook last week. Writes short, polite messages.", "goal": "Have the order of a cookbook delivered to my new address.", "facts": {"order": "B-35", "postcode": "10275"}}}}

The customer gives the order number and the postcode only when asked. Three personas also set an opening, their first message word for word.

The suite

version: 1
project: support-desk
dataset: data/customers.jsonl
system:
  name: support-desk
  version: baseline
  callable: desk:talk
  # A model plays each case's persona; here the stand-in in user_server.py.
  simulated_user:
    provider: openai_compatible
    model: stand-in-user
    base_url: http://127.0.0.1:8797/v1
    max_turns: 6
evaluators:
  # The desk answered every message the customer sent.
  - type: conversation_completed
  # The customer said they got what they came for.
  - type: goal_reached
  # ... within three of their own messages.
  - type: turns_to_goal
    max_turns: 3
  # No answer promises what the shop cannot promise.
  - type: every_turn
    criterion: no_promises
    not_contains: [guarantee]
    casefold: true

The stand-in simulated user

A simulated user is a model that plays each persona. Here it is user_server.py, which speaks the same API on 127.0.0.1 and is not a model: it reads the persona from the prompt Oloproof sends, states its goal, gives its order number or postcode when asked, leaves when the desk cannot help, and, asked whether its goal was met, says yes when the desk's last answer starts with "Done:". Start it in a second terminal and leave it running:

python user_server.py --port 8797

The policy

version: 1
confidence_level: 0.95
block_on: [FAIL, INSUFFICIENT_EVIDENCE, MANUAL_REVIEW]
require_validated_evaluators: true
minimum_simulated_user_realism: 0.80
rules:
  - id: no-promises
    metric: no_promises
    kind: observed_count
    max_failures: 0
  - id: resolved
    metric: goal_reached
    min: 0.80

No answer may promise. At least 80% of customers must reach their goal, decided only when the interval's lower bound clears 80%. And no rule decides on simulated customers until people have found at least 80% of their conversations realistic, by the same kind of lower bound.

Run it

oloproof run

Gate: BLOCK (exit 3)
│ no-promises │ no_promises  │ INSUFFICIENT_EVIDENCE │ simulated_user_not_validated │
│ resolved    │ goal_reached │ INSUFFICIENT_EVIDENCE │ simulated_user_not_validated │
│ conversation_completed │ 100.0%   │ [88.4%, 100.0%] │ 30 / 30 observed · 0 missing · 0 excluded │
│ goal_reached           │ 73.3%    │ [54.1%, 87.8%]  │ 22 / 30 observed · 0 missing · 0 excluded │
│ goal_within_3_turns    │ 73.3%    │ [54.1%, 87.8%]  │ 22 / 30 observed · 0 missing · 0 excluded │
│ no_promises            │ 60.0%    │ [40.6%, 77.4%]  │ 18 / 30 observed · 0 missing · 0 excluded │

Every customer was answered, 22 of 30 reached their goal, and 12 conversations contain a promise. Nothing is decided yet: these customers are simulated, and nobody has checked that they behave like real ones.

Read the simulated customers

oloproof simulated-users RUN_ID

  Realism: no conversation has been drawn and labelled yet.
  Not validated: nothing labelled yet, against a bar of 80.0%.
Features (median [quartiles]; descriptive, never a gate):
  user messages           2 [1, 2]
  user message length     49 [10, 67]
  How the simulated conversations ended: goal_reached 73%, user_left 27%

The eight who left are the customers who had moved: the baseline cannot change an address.

Label the conversations for realism

Draw a random sample of the run's conversations into a sheet, have a person read each one, with the persona and how it ended, and mark whether a real customer could have written the customer's side and ended it there, that way. Here fill_realism.py stands in for that person, copying their verdicts from labels/realism_verdicts.csv:

oloproof labels export RUN_ID --criterion simulated_user_realistic --sample 30 --local --out realism.csv
python fill_realism.py realism.csv
oloproof labels import realism.csv
oloproof simulated-users RUN_ID

  Realism: 29 of 30 drawn conversations labelled realistic (30 labelled), [82.8%, 99.9%] at 95%
  Validated: the lower bound clears the bar of 80.0%.

The person found one conversation unrealistic: customer c05 opens with "Order: F-15. Refund.", which reads like a form, not a person. With 29 of 30, the exact lower bound is 82.8%, above the bar, so rules may now decide on these customers.

oloproof gate RUN_ID --policy release.yaml

no-promises: FAIL (observed_failures_exceed_limit)
resolved: INSUFFICIENT_EVIDENCE (interval_overlaps_threshold)
Gate: BLOCK (exit 1)

Make a real change

desk_v2.py changes an address once it has the order and the new postcode, and says it has started a refund that "usually arrives within 5 days" instead of promising it. Copy it over desk.py, set version: candidate under system in oloproof.yaml, keep the stand-in running, and:

oloproof run

Gate: ALLOW (exit 0)
│ no-promises │ no_promises  │ PASS  │ observed_failures_within_limit │
│ resolved    │ goal_reached │ PASS  │ lower_bound_meets_minimum      │

The candidate met the same simulated customers on the same suite, so the realism measured on the baseline's conversations covers it: one validation covers one simulated user on one suite, whichever system it talked to.

Compare the candidate with the baseline

oloproof compare CANDIDATE_RUN_ID BASELINE_RUN_ID --policy compare.yaml

goal_reached: +26.7 points [+1.2, +54.1] · 30 paired · 0 missing · 0 excluded
  resolves-more  goal_reached  superiority  PASS  difference_above_zero

Each customer met both desks, so the comparison is paired customer by customer.

Simulated beside real

Realism labels judge each conversation alone. Setting the simulated customers beside real ones shows what a sample can miss. real.yaml replays 20 chats the shop's human agents had with real customers:

oloproof run --config real.yaml --policy logged_release.yaml
oloproof simulated-users BASELINE_RUN_ID --real REAL_RUN_ID

                          simulated (run_01M52Q1C)    real (run_01M52Q25)
  user messages           2 [1, 2]                    3 [3, 4]
  user message length     49 [10, 67]                 13 [4, 32]

Real customers sent more and shorter messages: they say what they want over three lines, where the stand-in says it in one tidy sentence. That is worth knowing about any simulated user, and it never decides anything here: comparing two unpaired samples needs a method Oloproof has not admitted.

Optional: a real model as the customers

This step leaves the offline path. It needs a chat model: locally, Ollama (ollama pull qwen2.5vl, or any chat model), no key and no cost; in the cloud, provider: anthropic or openai with api_key_env, billed per message. In oloproof.yaml, point the simulated user at it:

  simulated_user:
    provider: openai_compatible
    model: qwen2.5vl
    base_url: http://localhost:11434/v1
    max_turns: 6

Ollama answers one request at a time, so set concurrency: {system: 1} and a timeout_s of a few minutes. A different model is a different simulated user: the realism labels above do not cover it, so its conversations need their own sample before any rule decides on them.

Ten of the personas, played by qwen2.5vl on this machine against the candidate desk, reached their goal in 8 conversations, where the stand-in reached it in all 30. The model found what the stand-in could not: two customers asked for the "status" and the "shipping status" of their order, words the desk does not know, and got its greeting back each time. One ran out of messages. The desk understood the other's third try, and that customer left anyway: the goal check, a model's judgement, missed the answer. A realism label is where that shows: a real customer told when their order arrives does not leave unhappy. Unlike the stand-in, the model also often gave its order number before being asked, and wrote one longer message where the stand-in wrote two short ones.

Troubleshooting

Symptom	Cause and fix
Every rule is INSUFFICIENT_EVIDENCE with simulated_user_not_validated	Nobody has labelled this simulated user's conversations on this suite, or the realistic share does not clear the bar. Label a sample as above.
a case for a simulated user needs a 'persona' object in its input	A case has no persona, or it is under another key: set persona_field.
provider connection failed from every case	The stand-in is not running, or listens on another port than base_url names.
the simulated user's goal check answered ...	The model did not answer the goal check with yes or no. Try another model, or a larger one.
conversation/v1 is written by Oloproof	The desk recorded the conversation itself. Remove that: Oloproof records what passed through the simulated user.

Limitations

The stand-in customer is a rule. Its conversations show the mechanics; a real model's are messier, as the optional step shows.
Whether a goal was met is the simulated user's model's judgement, checked only through the realism labels.
A hosted workspace does not decide on simulated users yet.
Simulated customers are not real ones: a validated simulated user is evidence about customers like its personas, as far as people found it realistic, and no further.