Guides
Reinforcement learning
Oloproof evaluates a reinforcement-learning policy by running it in episodes, or from the log of the policy it would replace. Run it when you can; use the log when the outcome only happens in the world. This page is the reference for both, and the tutorial walks through each on a runnable example.
Oloproof does not host, reset or simulate an environment, and it does not train policies. Your system runs its own environment, as every system it evaluates runs its own code.
Episodes
A task is a case. An episode is one measurement of it: your system runs the policy in its environment and records the episode with current_case().rl_episode(...), as an rl_episode/v1 artifact.
| Field | Type | Meaning |
|---|---|---|
| steps | list of steps, index from 1 | Each step's reward, its constraint cost (default 0), and optionally action and observation |
| success | true, false or absent | The environment's own verdict on the episode |
| truncated | boolean, default false | The step limit ended the episode before a terminal state |
| horizon | integer, optional | The step limit; the episode may not be longer |
The record holds every step of the episode. A truncated episode is one the evaluation protocol defines as ending at its limit, so its return and cost are complete, not lower bounds.
Declare the kind the system records, so a run can refuse an evaluator that needs a record the system never writes:
system:
name: grid-agent
callable: env:run_baseline
records: [rl_episode/v1]Episode evaluators
| Type | Measures | Fields |
|---|---|---|
| episode_return | The return, each reward weighted by discount per step, as a score | return_range (required), discount (default 1) |
| episode_success | Pass when the episode records success: true; missing when it records none | none |
| episode_cost_within | Pass when the steps' costs add up to at most budget | budget (required) |
return_range is the range every return lies in, such as 30 steps at -1 each at worst and the goal reward at best. The interval on a mean return holds only inside it, so a return outside it stops the run with exit 2. In the SDK the classes are EpisodeReturn, EpisodeSuccess and EpisodeCostWithin in oloproof.evaluators.
Several episodes per task
Set replicates: to run several episodes of every task. current_case().replicate says which one is running (0 for the first), and it is part of the stored execution's key, so an environment that seeds the episode from it replays the same episode for the same replicate and a second run reuses what it stored.
The task stays the unit. Each criterion becomes the mean over tasks of each task's own mean over its episodes, and the interval counts tasks: five episodes of one task are not five independent tasks.
A pass rate over replicated tasks reads each task's share of passing episodes. To ask instead whether a task passes every time, declare an every_replicate metric:
metrics:
- {id: solved_every_time, type: every_replicate, criterion: episode_success}A task counts only when all of its episodes passed, so each task is one pass or fail and the exact binomial interval applies. With one episode per task it equals the pass rate. A task with a missing episode is missing, even when another of its episodes failed.
Comparing two policies
oloproof compare pairs each task with itself, each side worth its mean over its episodes. A return's difference is shown in the return's own unit, a pass rate's in percentage points. The two runs may run different numbers of episodes per task, except for an every_replicate metric: succeeding on every one of five episodes and on the only one are different questions, so that comparison is not computed. Quantile and ranking comparisons are not computed when either run has several episodes per task.
Off-policy evaluation
Use it when the outcome you care about only happens in the world (a ticket resolved, a user who came back) and you cannot try the new policy on real traffic yet. The evidence is a log of decisions an older policy made.
Each case is one logged decision:
{"id":"t001","input":{"topic":"technical","length":"short"},"expected":{"action":"small","logging_probabilities":{"small":0.7,"large":0.3},"reward":1}}| Field | Meaning |
|---|---|
| input | What the policy saw when it decided |
| expected.action | The action the logging policy took |
| expected.logging_probabilities | The probability the logging policy gave every action at that moment, adding up to 1 |
| expected.reward | What followed, inside reward_range; absent if it never arrived |
The system is the policy you want to evaluate. It receives input and returns its own probability of every action under probabilities, or the field you name in probabilities_field. It never runs a model or touches the world. input holds only what the logging policy saw when it decided: never the logged action or the reward, which the policy under test must not see. A policy that samples its choice returns the probabilities it samples from, not the action it drew; each decision is evaluated afresh even when its input repeats. Actions are named exactly as the log names them: an action spelled differently counts as one the logging policy never took.
| Type | Estimates |
|---|---|
| off_policy_value | The policy's mean reward on decisions like the logged ones |
| off_policy_improvement | That mean minus the logging policy's own |
Both take reward_range and propensity_floor, the smallest probability the logging policy gave any action the new policy may take. Set it from how the logging policy was designed (an exploration rate, a minimum share), never by trying values on the log. Each logged decision is reweighted by the new policy's probability of its action over the logging policy's, and the floor bounds that weight: without it no interval holds. In the SDK the classes are OffPolicyValue and OffPolicyImprovement.
What the engine does with a decision it cannot read as given:
| Situation | Result |
|---|---|
| Probabilities that do not add up to 1, a logged action with probability 0, a reward outside reward_range | The run stops with exit 2: a wrong probability would make the interval wrong without showing it |
| The new policy may take an action logged with probability below the floor | The decision is missing and bounded at its worst |
| The reward is absent | The decision is missing and bounded at its worst |
What the answer assumes:
- The probabilities were recorded when the logging policy decided, and they are the ones it used. Oloproof never estimates them.
- Each decision's reward depends on its own input and action, not on other decisions.
- The logging policy kept the same probabilities over the window of the log.
- The answer describes decisions like the logged ones, over the period the log covers.
The interval is wide when the floor is small: the decisions that inform a rarely chosen action are few, and each counts many times over. A longer log, and a logging policy that explores more, narrow it.
What is not built
- Off-policy evaluation of a sequence of decisions, where each choice changes what comes next.
- A log whose policy changed its probabilities as it learned during the window.
- Probabilities estimated after the fact.
- Hosting or resetting environments and simulators.
Where to go next
- Tutorial: evaluate a reinforcement-learning policy: both parts, run end to end.
- Writing a suite: replicates and the other suite fields.
- Configuration reference: every field of these evaluators.