Images, audio and video

Oloproof evaluates systems that take or produce images, audio and video, and the files never leave the place your system keeps them. A run records each file by reference; deterministic checks measure the bytes; a judge can see images, video frames and audio; the workbench shows them beside each verdict. The tutorial walks through images on a runnable example.

Recording media

Your system records the files a case received or produced with current_case().media(...), as one media/v1 artifact per call:

from oloproof import current_case, media_file, system

@system(name="captioner", version="v2", records=("media/v1",))
def caption(case):
    image = render(case["prompt"])            # your system writes its own file
    current_case().media([
        media_file(case["photo"], role="input"),
        media_file(image),                    # role="output" by default
    ])
    return {"caption": describe(image)}

In oloproof.yaml, a callable system declares records: [media/v1]. Each item holds:

media_file(path) fills these from a local file. Nothing is copied into Oloproof's store. Everything that reads the file later, a check, a judge or the workbench, reads it through read_media, which refuses bytes whose size or sha256 differs from the record: changed content stops the run, and a file that is gone leaves the case missing. Keep a run's files, unchanged, for as long as you keep the run; write each version's outputs to their own place. In your own @evaluator, read_media(item) returns the checked bytes.

Checks without a model

media_check measures each recorded item of a role and passes when every one meets every condition declared:

  - type: media_check
    criterion: thumbnail_fits
    kind: image
    formats: [png, jpeg, webp]
    max_width: 256
    max_height: 256
    max_bytes: 50000

The format is read from the file's opening bytes, so a JPEG saved as .png fails; image sizes and WAV durations are read from their headers; a video's size or an MP3's length is measured with ffprobe. Duration bounds apply to audio and video, dimension bounds to images and video.

image_matches_reference passes when each image looks like a reference: their 64-bit perceptual hashes differ in at most max_distance bits (8 by default). A resized or recompressed copy stays within a few bits; a blank, a broken file or a different scene does not. The reference is one reference_file for every case, or a path each case's expected names with reference_field. The hash reads layout and light, not colour, so it cannot tell two products photographed the same way apart, and flat graphics with large blank areas move it more than photographs do. It needs Pillow: pip install 'oloproof[media]'.

Every field is in the configuration reference. Both checks decide from the first run, since they call no model.

Judges that see media

A rubric_judge with media is shown the case's recorded media of that role (input, output or all), each named in its prompt so the judge can tell an input from an output:

  - type: rubric_judge
    criterion: caption_matches
    provider: openai_compatible
    base_url: http://localhost:11434/v1
    model: qwen2.5vl
    rubric_text: Pass only if the caption truthfully describes the attached image.
    media: output
    video_frames: 4
    transcriber:
      model: whisper-1

• Images are attached as they are, up to 5 MB each and 20 per judgment.
• Video is attached as still frames, taken with ffmpeg on the machine that runs the judge: video_frames: 4 takes the middle of four equal spans, [0.5, 2.0] takes those moments.
• Audio is transcribed by the transcriber (an OpenAI-compatible /audio/transcriptions endpoint, OpenAI's when base_url is left out) and the transcript goes into the prompt; or, with audio: native, it is attached as sound for a model that hears (wav and mp3; not Anthropic's).

The media's digests are part of what a judgment is keyed on, so changed media is judged again and unchanged media comes from the cache. The transcriber and the frame times are part of the judge's version, and each judgment keeps its transcripts and the ffmpeg version. A judge declared for audio with no transcriber, or for video on a machine without ffmpeg, stops the run rather than leaving those cases missing. A hosted model bills each image as well as each word.

Validating a media judge

A judge that sees media decides nothing until people's labels show it agrees with them, exactly as one that reads text: label a random sample, then oloproof evaluators validate (Judges). In a hosted workspace, the review queue shows each case's media to the reviewer. Changing the model, the rubric, media, the frames or the transcriber makes a new judge, which needs its own validation.

Media in the workbench

A case's media appears beside its output and its verdicts, on the case's page and in the review queue, with each file's type, size and digest.

• A local workbench reads each file from where the run recorded it, on your machine, and shows it only while its sha256 still matches.
• A hosted workspace never receives media. The viewer's browser fetches each file from oloproof collect on your network, with the same short-lived ticket the review queue uses for case content (Judges), and checks its sha256 before showing it. Without a collector, the page lists each file's facts and says why it is not shown.

A pushed run carries the media/v1 records, with where each file is kept, under the egress category artifacts; a policy that withholds artifacts keeps those locations on your side too.

Not supported

• Generating media: your system makes its files, and Oloproof evaluates them.
• Uploading media to Oloproof: there is no upload, hosted or local.
• A model watching a whole video or hearing audio through Anthropic's API: video is judged from frames, and audio through a transcript there.
• Deterministic checks of what audio or video contains (word error rate, scene detection): write one with @evaluator and read_media.
• Perceptual hashing of audio or video: image_matches_reference reads images only.