Simulated users

A scripted conversation tests your assistant against a user who never reacts. A simulated user reacts: a model plays a person each case describes, writes each next message from what your assistant just said, and stops when its goal is met or it gives up. Oloproof records what passed between them, measures it like any conversation, and lets a gate decide on it only once people have found the simulated user realistic. The tutorial walks through a support desk talking with simulated customers.

Personas

Each case's input holds a persona: who the user is, what they want, and what they know.

{"id": "c25", "input": {"persona": {
  "description": "A retired teacher who ordered a cookbook last week. Writes short, polite messages.",
  "goal": "Have the order of a cookbook delivered to my new address.",
  "facts": {"order": "B-35", "postcode": "10275"}
}}}

• description and goal are required. Write a goal a reader could check from the conversation alone: "have the shop start a refund" can be decided from what the assistant says, "get a refund" cannot, because the money arrives later.
• facts are what the user knows and shares only when the conversation needs it, such as an order number. Your assistant never sees the persona.
• opening is the user's first message, word for word, when every system should be greeted the same way.
• max_turns caps this user's messages, overriding the project's.

A persona is dataset content: it is covered by the suite digest and is the same for every system measured against it.

Declaring the simulated user

The model that plays every persona is declared on the system, like a judge:

system:
  name: support-desk
  callable: desk:talk
  simulated_user:
    provider: openai_compatible
    base_url: http://localhost:11434/v1
    model: qwen2.5vl
    max_turns: 6

Every field is in the configuration reference. A local model costs nothing; a hosted one bills every message it writes, and its tokens are recorded beside the conversation and never counted as your system's cost.

Your system keeps its loop

Oloproof does not drive the conversation. Your system asks for the user's first message, answers it however it answers users, and hands each answer back for the next message:

from oloproof import current_case

def talk(case):
    user = current_case().simulated_user()
    message = user.opening()
    while message is not None:
        message = user.reply(my_assistant.answer(message))
    return {}

reply returns None once the user stops. Your system may end the conversation itself with user.end(answer), as a handoff to a person would; an async def system awaits aopening(), areply(answer) and aend(answer). Oloproof records what passed through the handle as conversation/v1 and how it ended as simulated_user/v1, and refuses either from your system (SDK reference).

How a conversation ends

After each of your system's answers, the simulated user's model is asked, as a separate question at temperature 0 and without the role-play around it, whether that answer accomplished the goal. If it did, the conversation ends goal_reached. Otherwise the user writes its next message, or leaves (user_left), until it has sent max_turns messages (turn_limit). A system that stops without answering the last message leaves the conversation unanswered and truncated.

Asking separately is what makes a small local model usable: playing a persona and deciding whether a goal was met are different tasks, and a model asked to do both in one reply tends to keep chatting after its goal was met, or to say it is leaving when it is satisfied.

Measuring the conversations

The conversation is the unit: each evaluator gives one verdict per conversation, so every interval, comparison and gate applies unchanged.

Replicates (replicates:) meet differently seeded users, so one persona is several conversations. The simulated user's version (its model, prompt, temperature and budget) is part of each execution's cache key, so a different simulated user measures your system again.

Validating the simulated user

A result over simulated users reaches real users only as far as the simulated user resembles them. So every rule over a run whose users were simulated is INSUFFICIENT_EVIDENCE with simulated_user_not_validated until people have found its conversations realistic:

oloproof labels export RUN_ID --criterion simulated_user_realistic --sample 30 --local --out realism.csv
oloproof labels import realism.csv
oloproof simulated-users RUN_ID

The sheet shows each sampled conversation with the persona and how it ended. A person marks passed yes when a real customer could have written the user's side and ended it there, that way, which covers the goal check as well: a conversation that ended goal_reached when the goal was not met is not realistic. The simulated user is validated on the suite when the exact lower bound of the realistic share clears minimum_simulated_user_realism in release.yaml. A sampled conversation left unlabelled counts as unrealistic, and one is realistic only when everyone who labelled it says so.

One validation covers one simulated user's version on one suite, whichever system it talked to, so a candidate measured against the same personas is decided on the baseline's labels. A new model, prompt setting or persona set needs new labels. Setting require_validated_evaluators: false decides on simulated users without them, knowingly.

Simulated beside real

oloproof simulated-users RUN_ID --real REAL_RUN_ID sets the shape of the simulated conversations beside a run of real ones (logged chats replayed as conversation/v1): how many messages the user sent, how long they were, how long the answers were, how often a conversation was cut short, and how the simulated ones ended. It is descriptive and never decides: comparing two unpaired samples needs a method Oloproof has not admitted. Read it for what realism labels miss, such as simulated users who write in one tidy sentence what real ones spread over three.

Not supported

• Deciding on simulated users in a hosted workspace: its decisions stay withheld until its review queue can draw and seal realism labels itself.
• A gate on how simulated conversations compare with real ones.
• Turn-level rates within a conversation, and replaying a conversation from a turn.
• Simulated users for an HTTP system: your system's own code runs the loop, so it must be a callable.