Tutorial: images, with checks and a judge that sees them

Evaluate a function that makes images, here a product thumbnailer that also writes a caption. Check the thumbnails with measurements that need no model, judge the captions with a model that sees each thumbnail, measure that judge against a person, then ship a real change. Every step runs on this machine with no model and no network, and an optional step swaps in a real vision model.

What you will build

A catalogue service takes a product photo, saves a thumbnail, and writes a one-line caption naming the product's colour. Three questions decide whether a version may ship:

• Does each thumbnail fit the catalogue: a web format, at most 256 by 256 pixels, at most 50 kB? That is a measurement, made by media_check from the thumbnail's bytes.
• Is each thumbnail still the product's photo, and not a blank, a crop or another product? image_matches_reference compares it with the photo by perceptual hash.
• Does each caption name the colour a shopper sees? That needs eyes, so a rubric judge is shown the thumbnail.

The photos and thumbnails stay in the project folder. The system records them by reference (media/v1): where each file is, its type, size and sha256. Terms such as case, run, judge and gate are defined in Concepts; everything about media is in Images, audio and video.

Prerequisites

• Python 3.11 or later, and Oloproof with its media extra, which adds Pillow for comparing images:

python3 -m venv .venv
. .venv/bin/activate
pip install 'oloproof[media]'

• The example project, which ships with the package. Copy it into a new directory and work there:

oloproof init --example media product-thumbnails
cd product-thumbnails

• Port 8798 free for the stand-in judge (change it in both places if not).

Every step until "Optional: a real vision model" is offline and deterministic: no API key, no provider account, no cost.

The files

product-thumbnails/
  app.py                              the thumbnailer under test (baseline)
  app_v2.py                           the candidate change
  judge_server.py                     a stand-in vision judge speaking the OpenAI API on 127.0.0.1
  rubrics/caption_names_colour.md     the judge's rubric
  oloproof.yaml                       the suite
  release.yaml                        rules for a run
  compare.yaml                        a rule for a comparison
  data/products.jsonl                 20 cases
  photos/                             the 20 product photos the cases name
  make_photos.py                      how the photos were drawn
  labels/reviewer_verdicts.csv        one person's verdicts on the baseline's captions
  fill_labels.py                      copies those verdicts into a labelling sheet

Run every command from product-thumbnails/.

The application records its images

# app.py (excerpt)
@system(name="thumbnailer", version=VERSION, records=("media/v1",))
def describe(case: dict[str, Any]) -> dict[str, str]:
    photo_path = HERE / case["photo"]
    photo = Image.open(photo_path).convert("RGB")
    thumb = photo.copy()
    thumb.thumbnail((320, 320))
    out = HERE / "thumbs" / VERSION / photo_path.name
    out.parent.mkdir(parents=True, exist_ok=True)
    thumb.save(out)
    given = media_file(photo_path, role="input")
    made = media_file(out, width=thumb.width, height=thumb.height)
    current_case().media([given, made])
    centre = photo.getpixel((photo.width // 2, photo.height // 2))
    return {"caption": f"A {colour_name(centre)} product."}

media_file(path) describes a file: its file:// location, MIME type, size and sha256, named after the file. role="input" marks the photo the system was given; the thumbnail is an output, the default. current_case().media(...) records both as one media/v1 artifact, and records: [media/v1] in oloproof.yaml declares that the system records it. Nothing is copied: a check or a judge reads each file back later and refuses one whose bytes no longer match the record.

This baseline has two faults. Its thumbnails are 320 pixels wide, and its caption names the colour of the photo's centre pixel, which is the backdrop whenever the product sits to one side. Each version writes its thumbnails under thumbs/<version>/, so a later run never overwrites the files an earlier run recorded.

The dataset

{"id": "p03", "input": {"photo": "photos/p03.png"}, "expected": {"photo": "photos/p03.png", "colour": "yellow", "shape": "triangle"}}

input is what the system receives. expected.photo is the reference the thumbnail is compared with; colour and shape are there for a person reading the case, and no evaluator reads them.

Choosing the evaluators

evaluators:
  # The catalogue's limits, measured from the thumbnail's bytes: a web format, at most 256 by
  # 256 pixels, at most 50 kB.
  - type: media_check
    criterion: thumbnail_fits
    kind: image
    formats: [png, jpeg, webp]
    max_width: 256
    max_height: 256
    max_bytes: 50000
  # The thumbnail is still the product's photo: a perceptual hash within 10 of 64 bits.
  - type: image_matches_reference
    criterion: same_photo
    reference_field: photo
    max_distance: 10
  # The caption, judged by a model that sees the thumbnail.
  - type: rubric_judge
    criterion: caption_names_colour
    provider: openai_compatible
    model: stand-in-vision
    base_url: http://127.0.0.1:8798/v1
    rubric_file: rubrics/caption_names_colour.md
    media: output

media_check measures each output image: the format from its opening bytes, the size from its header, never from what the system declared. image_matches_reference hashes the thumbnail and the photo expected.photo names, and passes when the hashes differ in at most 10 of their 64 bits; a resize or a recompression moves a hash a few bits, another picture moves it many. Neither calls a model, so both decide from the first run.

media: output attaches the case's output images to the judge's request, each named in its prompt. A judge decides nothing until people's labels show it agrees with them, so the third rule waits for that.

The stand-in judge

A rubric judge sends its prompt and the attached images to a model and reads back {"pass": true|false, "rationale": "..."}. Here the model is judge_server.py, which speaks the same API on 127.0.0.1 and is not a model: it decodes the attached thumbnail, finds its most common colour other than the backdrop's white and grey, and passes a caption that names it. Start it in a second terminal and leave it running:

python judge_server.py --port 8798

The policy

version: 1
confidence_level: 0.95
block_on: [FAIL, INSUFFICIENT_EVIDENCE, MANUAL_REVIEW]
require_validated_evaluators: true
rules:
  - id: thumbnails-fit
    metric: thumbnail_fits
    kind: observed_count
    max_failures: 0
  - id: same-photo
    metric: same_photo
    kind: observed_count
    max_failures: 0
  - id: caption-floor
    metric: caption_names_colour
    min: 0.80

Every thumbnail must fit and must be its photo. At least 80% of captions must name the right colour, decided only when the interval's lower bound clears 80%.

Run it

oloproof run

Gate: BLOCK (exit 1)
│ thumbnails-fit │ thumbnail_fits       │ FAIL                  │ observed_failures_exceed_limit   │
│ same-photo     │ same_photo           │ PASS                  │ observed_failures_within_limit   │
│ caption-floor  │ caption_names_colour │ INSUFFICIENT_EVIDENCE │ evaluator_not_validated          │
│ thumbnail_fits       │ 0.0%     │ [0.0%, 16.9%]   │ 0 / 20 observed · 0 missing · 0 excluded  │
│ same_photo           │ 100.0%   │ [83.1%, 100.0%] │ 20 / 20 observed · 0 missing · 0 excluded │
│ caption_names_colour │ 60.0%    │ [36.0%, 80.9%]  │ 12 / 20 observed · 0 missing · 0 excluded │

No thumbnail fits, every thumbnail is its photo, and 12 of 20 captions name the right colour. The judge's rule is undecided because nobody has measured the judge yet.

Inspect the failures

oloproof inspect RUN_ID --failures

p03
  output: {"caption": "A white product."}
  thumbnail_fits: failed
    measured: p03.png: width 320 px is above 256 px
  caption_names_colour: failed
    judge text, not verified: the product is yellow; the caption says 'a white product.'

A check's reason is a measurement, marked measured; a judge's is its own words, which nobody has checked. The photos and thumbnails also appear beside each verdict on the case's page in the workbench, after oloproof export RUN_ID, read from this folder; a hosted workspace has your browser fetch them from oloproof collect and never holds them (Images, audio and video).

Measure the judge against a person

Draw a random sample of the run's cases into a sheet, without the judge's verdicts, have a person look at each thumbnail and caption, and import what they said. Here fill_labels.py stands in for that person, copying their verdicts from labels/reviewer_verdicts.csv:

oloproof labels export RUN_ID --criterion caption_names_colour --sample 20 --local --out sample.csv
python fill_labels.py sample.csv
oloproof labels import sample.csv

Then measure the judge against those labels, and record who did:

oloproof evaluators list
oloproof evaluators validate EVALUATOR_ID --by alice

caption_names_colour: sha256:53605dd6... is now VALIDATED
  agreement 100.0% [83.1%, 100.0%] · 20 of 20 labelled cases agreed · 0 labelled but not judged · kappa 1.000

A judge that sees images is measured exactly as one that reads text. In a hosted workspace the review queue shows each case's images to the reviewer, fetched by their browser from your collector (Judges).

oloproof gate RUN_ID --policy release.yaml

thumbnails-fit: FAIL (observed_failures_exceed_limit)
same-photo: PASS (observed_failures_within_limit)
caption-floor: INSUFFICIENT_EVIDENCE (interval_overlaps_threshold)
Gate: BLOCK (exit 1)

The judge may now decide, and 60% is not enough.

Make a real change

app_v2.py makes 256-pixel thumbnails and names the photo's most common colour other than the backdrop, wherever the product is. Copy it over app.py, set version: main-colour under system in oloproof.yaml, keep the judge running, and:

oloproof run

Gate: ALLOW (exit 0)
│ thumbnails-fit │ thumbnail_fits       │ PASS  │ observed_failures_within_limit │
│ same-photo     │ same_photo           │ PASS  │ observed_failures_within_limit │
│ caption-floor  │ caption_names_colour │ PASS  │ lower_bound_meets_minimum      │

The judge is the same evaluator version that was validated, so its verdicts count.

Compare the candidate with the baseline

oloproof compare CANDIDATE_RUN_ID BASELINE_RUN_ID --policy compare.yaml

thumbnail_fits: +100.0 points [+52.9, +100.0] · 20 paired · 0 missing · 0 excluded
caption_names_colour: +40.0 points [+1.8, +72.7] · 20 paired · 0 missing · 0 excluded
  captions-better  caption_names_colour  superiority  PASS  difference_above_zero

Optional: a real vision model

This step leaves the offline path. It needs a model that reads images: locally, Ollama with a vision model (ollama pull qwen2.5vl), no key and no cost; in the cloud, provider: anthropic or openai with api_key_env, billed per image as well as per word.

# live_judge.yaml
type: rubric_judge
criterion: caption_names_colour
provider: openai_compatible
model: qwen2.5vl
base_url: http://localhost:11434/v1
rubric_file: rubrics/caption_names_colour.md
media: output
timeout_s: 300

oloproof evaluators try live_judge.yaml

That judges the cases your reviewer labelled with the real model and reports its agreement, without adopting it. To adopt it, put it in oloproof.yaml in place of the stand-in: a new model is a new evaluator version, so validate it against the labels as above.

Troubleshooting

Limitations

• The stand-in judge counts pixels. It demonstrates the workflow, not judging quality.
• The perceptual hash reads layout and light, not colour: it tells a thumbnail of the photo from a blank or a different scene, not one product from another in the same studio. Flat graphics with large blank areas move it more than photographs do.
• Twenty labels give a wide agreement interval. Label more, at random and blind, for a judge you rely on.
• Audio and video work the same way with other settings: see Images, audio and video.