Configuration reference

Every field of oloproof.yaml and release.yaml, with its type, its default, the values it accepts and an example, taken from the models that read the files. Use it to look up a field; read the quickstart and gating pages to learn the workflow.

Both files are validated before anything runs. An unknown field, a misspelled field or a value of the wrong type is a configuration error and the command exits 2 without executing a case. Both files have JSON Schemas, which an editor that reads JSON Schema can use for completion. The installed package writes them, with the result schemas, into schemas/v1/ under the current directory: python -m oloproof_core.models.schema_export (the two are project_config.schema.json and release_policy.schema.json).

In the tables below, "required" means the file is refused without the field; any other field shows the value used when it is left out.

oloproof.yaml at a glance

A small, complete project. It runs a Python function locally, needs no network and no keys, and is the shape oloproof init scaffolds.

# oloproof.yaml
version: 1
project: support-bot
dataset: datasets/support.jsonl
system:
  name: support-bot
  callable: app.bot:answer
evaluators:
  - type: exact_match
    criterion: correct_label
    field: label

Top-level fields

concurrency

pricing entries

Oloproof ships no price table. Each entry names a model exactly as an evaluator's model: names it.

# oloproof.yaml
version: 1
project: support-bot
dataset: datasets/support.jsonl
system:
  name: support-bot
  callable: app.bot:answer
evaluators:
  - type: exact_match
    criterion: correct_label
    field: label
pricing:
  - model: my-judge-model
    input_per_mtok: 0.15
    output_per_mtok: 0.6
egress: [raw_outputs]

system

A system needs exactly one of callable, http or rag.

system.http

# oloproof.yaml
version: 1
project: support-api
dataset: datasets/support.jsonl
system:
  name: support-api
  version: "2026-10-08"
  http:
    url: http://localhost:8000/answer
    output_path: answer
    artifacts:
      retrieval/v1: debug.retrieval
evaluators:
  - type: hit_rate
    k: 5

The request and response contract, and what happens on timeouts and HTTP errors, are in Results and execution.

system.rag

Settings given here override the ones the class declares. A staged system records its own retrieval/v1, context/v1 and citations/v1 artifacts, so records is refused beside it. See RAG.

evaluators

Every entry takes a type and these two common fields:

fail applies only to pass/fail evaluators; a score evaluator with it is a configuration error. on_execution_error is a YAML field; the SDK evaluator classes take no such argument, and an errored case counts as missing.

Evaluator types

"Reads" lists what the evaluator's verdict depends on, which is also what its cached judgment is keyed on. "SDK" names the class in oloproof.evaluators.

A judge that calls a hosted model sends case content to that provider and is billed by it. Keys are read from the environment variable named in api_key_env; Oloproof never stores them in these files.

Deterministic evaluators

Model judges

rubric_judge, groundedness_judge and citation_support_judge share these fields. Exactly one of rubric_file or rubric_text is required by rubric_judge; the two RAG judges take at most one and otherwise use a built-in rubric. Their criterion defaults to groundedness and citation_support.

probability_judge asks a typed question and reads the model's probabilities:

cascade runs a cheap judge first and escalates the uncertain cases:

The stages judge the cascade's own criterion; a stage naming a different one is refused.

model_classifier scores text with a trained model on a TEI-compatible server:

RAG evaluators

Agent evaluators

Each agent type has a default criterion, so it may be left out: its own type name, or one built from its setting (agent_steps_le_8, agent_tool_lookup_called, agent_handoffs_le_2). See Agents.

Predictive evaluators

A predictive evaluator that leaves positive, field or expected_field unwritten takes it from the predictive: block; a value it writes is kept.

predictive

metrics

Each evaluator criterion is already a metric. A metrics: entry adds one more, discriminated by type.

# oloproof.yaml
version: 1
project: support-bot
dataset: datasets/support.jsonl
system:
  name: support-bot
  callable: app.bot:answer
evaluators:
  - type: exact_match
    criterion: correct_label
    field: label
metrics:
  - id: latency_p95
    type: quantile
    source: latency_ms
    quantile: 0.95

release.yaml

The release policy: which rules decide, and which decisions block. Omitted settings keep their defaults, so a policy that names only its rules still blocks on FAIL, INSUFFICIENT_EVIDENCE and MANUAL_REVIEW.

# release.yaml
version: 1
rules:
  - id: label_accuracy
    metric: correct_label
    min: 0.8

rules

One list holds both kinds. A run rule takes exactly one of min, max or max_failures. A comparison rule names its kind and decides a difference between two runs; see Comparison rules.

families

# release.yaml
version: 1
warn_on: [INSUFFICIENT_EVIDENCE]
block_on: [FAIL, MANUAL_REVIEW]
rules:
  - id: label_accuracy
    metric: correct_label
    min: 0.8
    max_missing_fraction: 0.05
  - id: no_regression
    metric: correct_label
    kind: non_inferiority
    margin: 0.02

Artifact kinds

An artifact is a typed record a system writes beside its output, such as what it retrieved. A kind is a lowercase name with an optional version, matching ^[a-z][a-z0-9_]*(/v[1-9][0-9]*)?$. Evaluators that need an artifact name it, and a run whose system does not declare a required kind is refused before it starts, rather than counting every case as missing.

A callable system declares the kinds it records in records: (or @system(records=...)); an HTTP system in http.artifacts; a staged RAG system records its own.

Versions, cache keys and invalidation

Oloproof reuses work whose inputs have not changed, and decides what "unchanged" means from content digests. Each is computed by the engine and recorded with the run.

What Oloproof cannot see is yours to declare:

• An HTTP system's behaviour lives on the server. Change system.version whenever what is behind the URL changes, or an old cached output will stand for the new system.
• A callable's helper modules are hashed only when code_paths matches them. Without it, editing a helper does not change the version.
• A method or a callable object must declare a version, and the version must change when the object's state changes.
• A RAG index is identified by index_version; change it when the index is rebuilt.
• A model judge's identity is its settings, not the provider's weights. A provider updating the model behind the same name is not detected by the cache.
• A custom @evaluator hashes the module file that defines it, and its judgments are reused across runs only when it declares cacheable=True. Built-in rubric judges are cacheable; deterministic evaluators are recomputed, which is cheap.

Cached work lives in the project's local store, .oloproof/store.sqlite beside oloproof.yaml (or under OLOPROOF_HOME). Deleting the store discards every cache and every run. In a hosted workspace the engine does not reuse cached executions, judgments or analyses, because a push can write them; it recomputes.