Choosing among candidates

oloproof select picks the best of several versions of a system that meets every requirement, and says separately whether it is really ahead of the next one. Use it when you have, say, three agents, two prompts or four retrieval settings, and one set of requirements.

Why not pick the best numbers

Picking the cheapest candidate whose numbers clear the bars has two problems. A measured 91% against a 90% bar is one point above it, inside the noise of a thousand cases. And when you look at several candidates, the one that cleared by luck is the one you pick. So each candidate must first establish every rule on its own, with a stricter confidence level because several are looked at: with three candidates and a 95% policy, each is decided at 98.33% (1 - 0.05/3). Only the candidates that pass every rule are ranked, and a cheaper one can never trade a safety rule for its cost.

Declaring a selection

Write a selection file beside your project, and decide it under the project's release.yaml:

select:
  candidates:
    agent_a: run_01JC8Q2X7D
    agent_b: agent_b.yaml
    agent_c: agent_c.yaml
  objective:
    metric: cost_per_task
    direction: minimize

A candidate is a stored run id, or a project file that oloproof select runs first. A run it can reuse costs nothing new. The policy may hold only run rules, and early stopping must be off.

oloproof select select.yaml
oloproof select select.yaml --save

--save also writes the selection for the local workbench, under Candidates.

Reading the answer

There are three outcomes:

• A candidate is selected and its lead over the runner-up is established: the paired comparison of the two on the objective is PASS.
• A candidate is selected and its lead is not established: the runner-up would have been an equally defensible choice on this evidence.
• No candidate qualifies: each candidate's rules are listed with their answers, so you see what is missing.

The command exits 0 when a candidate is selected and 3 when none qualifies.

A worked example

Three agents must reach a success rate of at least 90% and a violation rate of at most 1%; the objective is the lowest cost per task.

Each is decided at 98.33%. At 1,000 cases no agent qualifies: A's violation rate is not yet shown to be under 1%, B's 91% is too close to 90% to be established, and C breaks the violation ceiling. The answer is "no candidate qualifies", with what each one is missing. At 3,000 cases A establishes both rules and is selected. B is cheaper, but its success is still not established, so it is never ranked: picking B on its numbers would ship a requirement nobody has shown.

Sharing a selection

oloproof select select.yaml --save keeps the selection for the workbench. oloproof push --selection ID sends it to the hosted workspace with each candidate's run, through the project's egress policy; ID is the folder name under .oloproof/bundles/ or the selection's digest. Before it keeps the selection, the workspace makes it again from the runs it holds, under the policy those runs were gated with, and keeps it only if the result is the same: one whose result or words were edited, that names a run the project does not hold, or that the engine does not make from those runs, is refused, and nothing is stored. So gate the candidate runs under the selection's policy before you push them. The project's Candidates page lists it, and its page shows it exactly as the local workbench does.

An Owner or an Admin can make an evidence page of it from the selection's page: the outcome, the level, and each candidate's name, run id and rules at the corrected level, labelled "Reported by the developer", never case content.

Next

• Comparison rules covers the paired comparison the lead is decided by.
• Gating covers the rules each candidate is held to.