Managed runs

A managed run executes your suite on Oloproof's workers instead of your machine, and you watch it from the Jobs page or your terminal. You need only oloproof login; no database connection and no runner of your own.

What a worker will run

A worker calls your system over HTTP. Declare it with http: in oloproof.yaml (Configuration):

system:
  name: support-bot
  version: v3
  http:
    url: https://support-bot.example.com/answer
    output_path: answer

A worker refuses:

• A callable: or rag: system, before the job is queued. That is your code, and Oloproof's workers do not run customer code; run it on your own machine (oloproof run) and push the run, or on a runner you operate.
• A recorded: answers file. It stays on your machine; run it locally with oloproof run.
• An address that is not on the public internet: localhost, a single-label name, a private, link-local or cloud metadata address. Each call is checked as it is made, so such a case errors rather than reaching it; a worker can reach only what anyone on the internet can.
• Generated attack cases, unless oloproof.yaml declares adversarial: {sandboxed: true} (Generating attacks). --allow-adversarial covers a local run only.

Run one

oloproof login
oloproof run --cloud --max-runtime 900

run --cloud sends your suite's cases to the workspace (a managed run cannot execute cases it does not have), then queues the job and prints where to watch it. A suite the workspace already holds whole is not sent again.

The worker checks every case against its digest before it keeps the suite, and refuses one that does not match. So run --cloud digests the cases plainly, as the worker does, even in a project that keys its digests to withhold content (egress:, Results and execution): the cases are sent whole anyway, so keying them would hide nothing. The managed run's suite is therefore a different suite from the one a local run of the same project records. A project whose workspace already holds keyed evidence (from a push) is refused: run it with oloproof run and push the run.

--max-runtime is the job's deadline in seconds, up to a day, or your plan's limit if that is less. A budget: in oloproof.yaml binds on a worker as it does locally.

Who may run one: an Owner, Admin or Member of the workspace. A Reviewer may not.

Watch it

The project's Jobs page shows each job's cases as they arrive: how many have run and been judged, a pass and fail tally per check, and the latest cases. The tally is a count so far, not a result. The decisions are reached once the whole run is stored, and its page then shows them.

In the terminal:

oloproof job JOB_ID --follow

prints the same counts every few seconds, then the run's gate, and exits with the gate's exit code, so a pipeline can wait on a managed run as on a local one (CI). Without --follow, job reports once; a job still running exits 2, because it is not a decision yet. --cancel stops a queued job at once and a running one before its next call, keeping the cases that finished.

Keys for a judge

A judge or a system that calls a model provider needs its key on the worker. Store it once:

export OPENAI_API_KEY=...
oloproof credentials set OPENAI_API_KEY
oloproof credentials list

The key is read from the environment or standard input, never from the command line. It is encrypted by the worker and readable only by a worker running this workspace's jobs; nobody can read it back, and credentials list shows names only. An evaluator's api_key_env: OPENAI_API_KEY then finds it. A managed job reads no other environment variable. Only an Owner or Admin may store a key.

What this cannot tell you

• Whether your system behaves the same from Oloproof's network as from yours. A worker calls your endpoint from the public internet; rate limits, regions and allow lists that treat it differently change what it measures.
• Anything about code a worker did not run. A managed run measures the endpoint you named, not the code behind it.

Every other limit on What an evaluation cannot tell you holds here too.