Human review

Your own people can label your system's answers in Oloproof, and their labels can decide a release on their own or check the judge that decides it. This page gathers what exists and where each part is explained in full.

What people can label

• Pass, fail or can't tell on one criterion, case by case.
• A score on a scale you choose, such as 1 to 5.
• A preference between two runs' answers to the same case, shown as A and B in an order the workspace fixes by a recorded coin, so the reviewer cannot tell which run is which.
• Pass or fail on both answers to the same case, for comparing two versions by people's verdicts.

A rubric written for the queue is shown beside each case.

Where they label

In the browser. In a hosted workspace, an Owner or Admin opens a review queue from the project's Review page and assigns reviewers. Each reviewer sees the queues assigned to them and labels with the buttons or the keys P, F and C. A verdict from an account not assigned to the queue is refused, and each verdict is recorded under the reviewer's own account.

In the terminal. oloproof review RUN --criterion NAME --by YOU labels one case per line. With no other flag it checks the judge in one go: a blind sample sized by your bar, the judge measured against your labels, and the run decided again.

In a spreadsheet. oloproof labels export RUN --criterion NAME --sample 50 writes a CSV for people to fill in, and oloproof labels import records every filled-in row or none of them. Use it for reviewers who work outside Oloproof, including a labelling vendor. Labels from a file count on your machine, not in a workspace's own check of a sample it drew.

Running a queue

When an Owner or Admin opens a queue, each reviewer is emailed that they were assigned, unless they turned review emails off. A queue can carry a due date, shown on the queue and in the email; it decides nothing, and nothing is sent when it passes. On the queue's page, Owners and Admins see each reviewer's progress (cases done and their last verdict) and can change the due date.

A review queue's reviewers can be changed at any time, and each one added is emailed. A measurement queue's reviewers cannot: they are part of its random sample, and only their labels count on it, so other reviewers take a new queue with its own sample.

Two kinds of queue

A measurement queue has the workspace draw a random sample and hides the judge's verdict. Its labels measure the judge, or decide a rule by people alone. A review queue of flagged cases shows the judge's verdict and settles those cases without measuring anything. Only the first kind can stand for the whole suite.

When reviewers disagree

A project can require two or more reviewers per sampled case, counted by account. Where they disagree on a pass or fail, the workspace decides nothing until an Owner or Admin who labelled none of that case resolves it, on the queue page or the run page. How often people agree with each other is shown beside each evaluator, as a share of pairs of reviewers who agreed, with its interval, and as Krippendorff's alpha, which corrects that share for the agreement chance alone would give on a criterion most answers pass: 1 is perfect agreement and 0 is chance. Alpha is given with an interval from 300 cases labelled by two or more people; below that it is shown alone, with the reason. It decides nothing.

A pass or fail measurement queue with three or more reviewers can instead be read by majority, chosen when it is opened. Each reviewer's first verdict is their vote; a case counts once more than half of its reviewers agree, and a tie waits for an Owner or Admin to resolve it. A run's pass or fail queues on one criterion are all read the same way.

Scores, preferences and pass or fail on two runs need no such choice: each case's value is the mean over every reviewer who labelled it, so a disagreement moves the value instead of waiting to be resolved, and a harsh or lenient reviewer is part of what the answer measures.

Labels from a vendor

A pass or fail or score measurement queue can be labelled by a vendor's raters instead of members. Open it with "Labelled by a vendor" and the raters' names. On the queue's page, download the sheet (a row per drawn case and rater, to fill in with pass or fail, or a score on the queue's scale), give it to the vendor with the answers to show, and upload the returned CSV or XLSX. Every row is checked: a case the queue did not draw, a rater it does not name, or a second verdict by one rater on one case refuses the whole file.

Each rater then counts as a reviewer by name, so agreement, alpha, accuracy on answer keys and the majority all read them. Key a few cases first to check a vendor's raters before you rely on them; their accuracy is reported and decides nothing. The workspace knows the file it was given, not who looked: each label names the account that uploaded it.

Answer keys and reviewer accuracy

An Owner or Admin can key some cases of a pass or fail queue: open a case and key it as pass or fail. Only an Owner or Admin the queue does not assign sees the keys, so reviewers are not told which cases are keyed. The Review page then shows, to Owners and Admins, each reviewer's share of keyed answers that matched the key, with its interval, and the same by month.

A reviewer's first pass or fail on the keyed answer counts; "can't tell" is shown beside it, resolutions and the key's author's own verdicts are left out, and a key is about one answer, so a verdict on another run's answer to the same case is not compared. A key is never a label and decides nothing. The interval reads the keyed cases as a random sample of cases like them; keys chosen by hand are not one, so read it as accuracy on these cases.

People's labels deciding a release

• human_score: what people score the whole suite, from the cases they scored.
• human_preference: whether people prefer the candidate to the baseline, and by how much.
• human_difference: by how much people pass the candidate more often than the baseline.
• judge_bias_bar: a comparison decided by a judge goes to a person unless people's labels show the judge treats both versions alike.

Each is declared in release.yaml or compare.yaml and explained, with its method and what it cannot claim, in Judges. A case nobody labelled counts at its worst value, so leaving the hard cases unlabelled never helps.

Not built yet

These do not exist today:

• a vendor labelling a review queue, preferences, or pass or fail on two runs: each of those shows a reviewer two answers in an order the workspace fixes per reviewer, which a file made outside Oloproof cannot keep.

Oloproof does not supply reviewers or connect to a vendor's own tools.