Tutorial: evaluate a reinforcement-learning policy

A runnable walkthrough for two kinds of reinforcement-learning question: how well a policy you can run performs over episodes, and how well a policy you cannot try in the world would have done, judged from the log of the policy it would replace. Both examples run locally with no provider credentials.

The reference for every field and evaluator is Reinforcement learning; the terms case, evaluator, metric, interval and gate are in Core concepts. This page is the hands-on route through them.

What Oloproof does and does not do here

Oloproof does not run your environment, host a simulator or train a policy. Your system runs each episode in its own environment and records it as an rl_episode/v1 artifact, and every episode metric is read out of that record. In the second part nothing runs at all: the policy under test only states the probability it would give each action.

Keep the two questions apart:

Prerequisites

• Python 3.11 or later, and Oloproof installed (pip install oloproof).
• The example project, which ships with the package. Copy it into a new directory:

oloproof init --example rl my-rl

It holds two projects, episodes/ and off_policy/. Every command below runs from inside one of them, and evidence is stored in its .oloproof/ directory.

Part 1: a policy you can run

cd my-rl/episodes

The files

An episode moves one cell per step until it reaches the goal or runs out of steps. Every step earns -1 and reaching the goal earns +10, so a return lies between -30 (30 steps, no goal) and 10. Entering a hazard costs 1: that is the safety constraint. One move in ten slips in a random direction, whatever the policy chose, so the same policy can do differently on the same grid.

Recording an episode

The system runs the episode itself and hands the record to the case recorder:

@system(name="grid-agent", version="wander-v1", records=("rl_episode/v1",))
def run_baseline(task):
    return _run(task, wander)


def _run(task, policy):
    recorder = current_case()
    record = episode(task, policy, seed=task["seed"] * 1000 + recorder.replicate)
    recorder.rl_episode(record)
    return {"success": record["success"], "steps": len(record["steps"])}

The record lists each step's reward and cost, whether the episode reached the goal (success), and whether the step limit ended it (truncated).

Several episodes per task

oloproof.yaml declares replicates: 5, so every task runs five episodes. current_case().replicate says which one (0 to 4), and the environment seeds the episode from it: the same episode of the same task always replays the same way, and a second run reuses the stored episodes instead of running them again.

The task stays the unit. Each criterion is the mean, over the 24 tasks, of each task's own mean over its five episodes, so the interval counts 24 tasks and not 120 episodes. Counting episodes would pretend five episodes of one grid are five independent grids.

The evaluators:

evaluators:
  - {type: episode_return, return_range: [-30, 10]}
  - {type: episode_success}
  - {type: episode_cost_within, budget: 0, criterion: hazard_free}
metrics:
  - {id: solved_every_time, type: every_replicate, criterion: episode_success}

return_range is required: the interval on a mean return holds only within a range every return lies in, and a return outside it stops the run. solved_every_time counts a task only when all five of its episodes reached the goal.

Run the baseline

oloproof run

Gate: BLOCK (exit 3)
                                        Decisions
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Rule          ┃ Metric          ┃ State                 ┃ Reasons                     ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ success-floor │ episode_success │ INSUFFICIENT_EVIDENCE │ interval_overlaps_threshold │
│ safety-floor  │ hazard_free     │ INSUFFICIENT_EVIDENCE │ interval_overlaps_threshold │
└───────────────┴─────────────────┴───────────────────────┴─────────────────────────────┘
                                                           Metrics
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Metric            ┃ Estimate ┃ Interval        ┃ N                                                                         ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ episode_return    │ -10.51   │ [-16.44, -5.11] │ mean of 24 observed · 0 missing · 0 excluded                              │
│ episode_success   │ 76.7%    │ [55.4%, 89.6%]  │ 24 cases observed · 0 missing · 0 excluded · 5 replicates each · 14 flaky │
│ hazard_free       │ 29.2%    │ [10.3%, 54.0%]  │ 24 cases observed · 0 missing · 0 excluded · 5 replicates each · 9 flaky  │
│ solved_every_time │ 37.5%    │ [18.7%, 59.5%]  │ 9 / 24 observed · 0 missing · 0 excluded · 5 replicates each · 14 flaky   │
└───────────────────┴──────────┴─────────────────┴───────────────────────────────────────────────────────────────────────────┘
Cache: execution 0 hit/120 miss; judgment 0 hit/360 miss

Read it row by row:

• The wandering policy reaches the goal on 76.7% of episodes, but the interval runs from 55.4% to 89.6%, so the floor of 75% is neither cleared nor missed: INSUFFICIENT_EVIDENCE.
• It solves only 9 of the 24 tasks on every one of its five episodes. "Usually works" and "works every time" are different claims, and an agent that will run unattended needs the second.
• 14 tasks are flaky: their success moved between identical episodes. One episode per task could not have shown that.
• It avoids hazards on 29.2% of episodes. The interval's top, 54.0%, is above the 50% floor, so even this poor record does not yet prove the policy unsafe.

Look at one task:

oloproof inspect RUN_ID --case grid_03

output: {
  "success": false,
  "steps": 30
}
judgments:
  episode_return: score -30
  episode_success: failed
  hazard_free: passed

The first episode of grid_03 ran out of steps: return -30, no goal, no hazard entered.

Run the candidate

oloproof run --config candidate.yaml

│ success-floor │ episode_success │ PASS                  │ lower_bound_meets_minimum   │
│ safety-floor  │ hazard_free     │ INSUFFICIENT_EVIDENCE │ interval_overlaps_threshold │

│ episode_return    │ 0.42     │ [-5.78, 2.59]   │ mean of 24 observed · 0 missing · 0 excluded                             │
│ episode_success   │ 100.0%   │ [79.9%, 100.0%] │ 24 cases observed · 0 missing · 0 excluded · 5 replicates each           │
│ hazard_free       │ 57.5%    │ [29.3%, 81.5%]  │ 24 cases observed · 0 missing · 0 excluded · 5 replicates each · 6 flaky │
│ solved_every_time │ 100.0%   │ [85.7%, 100.0%] │ 24 / 24 observed · 0 missing · 0 excluded · 5 replicates each            │

The planner reaches the goal on every episode, so the success floor passes. It still enters hazards: some grids have no safe path, and the slippery floor pushes it into others. Its safety rate of 57.5% is above the floor, but 24 tasks cannot yet show it is above 50%.

Compare the two policies

oloproof compare CANDIDATE_RUN_ID BASELINE_RUN_ID --policy comparison.yaml

episode_return: +10.92 [+0.21, +17.74] · 24 paired · 0 missing · 0 excluded
episode_success: +23.3 points [-3.2, +44.6] · 24 paired · 0 missing · 0 excluded
hazard_free: +28.3 points [-2.4, +54.7] · 24 paired · 0 missing · 0 excluded
solved_every_time: +62.5 points [+24.3, +87.7] · 24 paired · 0 missing · 0 excluded
Decisions
  return-better  episode_return  superiority  PASS  difference_above_zero
  safety-no-worse  hazard_free  non-inferiority, margin 5.0 points  PASS  lower_bound_above_margin
Gate: ALLOW (exit 0)

The comparison pairs each task with itself, each side worth its mean over five episodes. The return difference is in the return's own unit: the planner earns about 10.9 more per episode, and at least 0.21 more. Paired on the same grids, a 5-point margin is enough to show the planner is no less safe, even though neither run alone could settle its own safety floor.

Part 2: a policy judged from a log

cd ../off_policy

A support desk routes each ticket to a small or a large model. The current router picks the small model 70% of the time and the large one 30%, at random. You have written a new rule: technical tickets, and long billing tickets, go to the large model. You cannot try it on real tickets yet, but the old router's log can answer for it. That is off-policy evaluation.

The log and its files

Each line of the log is one decision:

{"id":"t001","input":{"topic":"technical","length":"short"},"expected":{"action":"small","logging_probabilities":{"small":0.7,"large":0.3},"reward":1}}

input is what the router saw, action what it chose, logging_probabilities the probability it gave every model at that moment, and reward whether the ticket was resolved. The new rule only answers with its own probabilities:

@system(name="ticket-router", version="by-topic-v1")
def route_by_topic(ticket):
    large = ticket["topic"] == "technical" or (
        ticket["topic"] == "billing" and ticket["length"] == "long"
    )
    return {"probabilities": {"small": 0.0 if large else 1.0, "large": 1.0 if large else 0.0}}

The evaluators:

evaluators:
  - {type: off_policy_value, reward_range: [0, 1], propensity_floor: 0.3}
  - {type: off_policy_improvement, reward_range: [0, 1], propensity_floor: 0.3}

Each logged decision is reweighted by how much more, or less, likely the new rule was to make it than the old router: a ticket the old router sent to the large model with probability 0.3, which the new rule would always send there, counts 1 / 0.3 times, and a decision the new rule would never have made counts zero. propensity_floor is the smallest probability the old router gave any model it could pick. It bounds that weight, and without a bound no interval holds.

Run it

oloproof run

│ resolution-floor    │ off_policy_value       │ PASS                  │ lower_bound_meets_minimum                                                          │
│ better-than-current │ off_policy_improvement │ INSUFFICIENT_EVIDENCE │ interval_overlaps_threshold                                                        │
│                     │                        │                       │ about 2633 more cases would decide it, if the observed values hold (3233 in total) │

│ off_policy_value       │ 0.775    │ [0.632, 0.921]  │ mean of 600 observed · 0 missing · 0 excluded │
│ off_policy_improvement │ 0.060    │ [-0.064, 0.188] │ mean of 600 observed · 0 missing · 0 excluded │

The new rule would resolve an estimated 77.5% of tickets and, at 95% confidence, at least 63.2%: the floor of 60% passes. Whether it beats the current router is not settled. The estimate is 6 points better, but the interval runs from 6.4 points worse to 18.8 better, and the engine says roughly how many more logged decisions would decide it.

That width is the price of learning from someone else's choices. Only the 30% of tickets the old router happened to send to the large model say anything about the large model, and each counts more than three times. Two things narrow it: a longer log, and an old router that explores more.

Logging decisions from your own system

• Record the probabilities when the router decides, not afterwards, and record every option's probability, not only the chosen one's. Oloproof does not estimate them: a guessed probability would make the interval rest on the guess.
• Put in input only what the router saw when it decided, never the chosen option or the outcome: the policy under test must not see them.
• If your new policy samples its choice, return the probabilities it samples from, not the one option it drew. Name each option exactly as the log does: an option spelled differently counts as one the old router never chose.
• Keep some randomness in production routing. A policy that never chooses an option leaves nothing in the log to judge a rule that would.
• The answer describes tickets like the logged ones, over the period the log covers. A change in your traffic is outside it.
• A malformed line (probabilities that do not add up to 1, a chosen option with probability 0, a reward outside reward_range) stops the run. A decision the new rule would make with an option the old router gave less than the floor, or whose reward is missing, counts as missing and widens the interval.

What is not built

• Off-policy evaluation of a sequence of decisions, where each step's choice changes what comes next. Only one decision per case is supported.
• A log whose router changed its probabilities as it learned during the window.
• Hosting or resetting environments, simulators or robots.

Where to go next

• Reinforcement learning: the reference for these evaluators and the episode record.
• Writing a suite: replicates and the other suite fields.
• Comparing a candidate to a baseline: what a paired comparison reads.
• Core concepts: what a result does and does not reach.