Skip to content
Docs
ChangelogPricingOpen console

Guides

Evaluating from Node

The Node SDK runs your system and its evaluators in Node, inside your own test runner, and hands the scores to the Python engine, which decides. A TypeScript team keeps its code, its fixtures and its test runner; every interval, comparison and decision stays the engine's, and the SDK computes none of them, not even an average.

Install

npm install --save-dev @oloproof/sdk
pip install oloproof

The first line is the SDK, the second the engine it hands the scores to (Python 3.11 or later). The SDK finds oloproof on your PATH; if it is not there, it stops and says how to install it.

Score a system in a test

A suite is the same JSON Lines file the engine reads (Writing a suite), one case per line, each with an id. Give evaluate the suite, the system, and one evaluator per criterion:

import { test } from "vitest";
import { type Case, decide, evaluate } from "@oloproof/sdk";
import { answer, rateHelpfulness } from "./support-bot.js";

type SupportCase = Case<{ question: string }, { label: string }>;
type Answer = { label: string; text: string };

test("the support bot is good enough to release", async () => {
  const scored = await evaluate<SupportCase, Answer>({
    suitePath: "data/support.jsonl",
    system: (testCase) => answer(testCase.input.question),
    evaluators: {
      exact_label: (output, testCase) => ({ passed: output.label === testCase.expected?.label }),
      helpfulness: {
        scale: [0, 1],
        judge: "gpt-4o-mini",
        evaluate: async (output, testCase) => ({ score: await rateHelpfulness(output, testCase) }),
      },
    },
    concurrency: 8,
    out: "scores/candidate.jsonl",
  });
  console.log(scored.counts);
}, 600_000);

An evaluator returns { passed } or { score }, with an optional reason. A function judges pass or fail; an evaluator that scores declares its scale, because an interval on a mean holds only inside a range known before the scores are read. Name the judge when a model judged: the engine then treats it as a judge to check against people before it may gate a release (Judges).

concurrency bounds how many cases run at once (four by default). The score file lists the cases in the order the suite gives them, whatever order they finish in.

Let the engine decide

const decided = await decide({
  scores: "scores/candidate.jsonl",
  baseline: "scores/baseline.jsonl",
  policy: "release.yaml",
});
console.log(decided.exit, decided.comparison?.gate?.decisions);

decide runs the commands you would type, in the directory given as cwd (the current one by default):

oloproof scores import scores/baseline.jsonl scores/candidate.jsonl --tool node
oloproof compare RUN_ID BASELINE_RUN_ID --policy release.yaml --json

With a baseline, the engine compares the two runs case by case and decided.comparison is the record oloproof compare --json printed. Without one, decide runs oloproof gate on the imported run and returns what it printed in decided.decision.stdout. Either way decided.exitCode is the engine's exit code, unchanged, and decided.exit its name: EXIT_CODES maps each name to the engine's value, as Running in CI lists them.

What the file holds

The file is the one oloproof scores import reads (A second opinion): one row per case and criterion, case ids and scores only. An answer, an input or a reason never enters it; they stay in scored.results, in Node.

  • A case whose system throws is written with an empty value on every criterion, and a case whose evaluator throws with an empty value on that criterion. The engine counts each as missing and bounds it at its worst, so a failure lowers what the evidence can show rather than disappearing from it. scored.errors lists each one with its message.
  • A verdict the file cannot carry stops the run with an error and writes nothing: a score with no scale or outside it, both a score and a pass, or a criterion the engine could not name as a metric.
  • Each file is a system version identified by its contents, so importing the same file again reuses what the engine stored.

Without installing the engine

secondOpinion sends two score files to the hosted page that answers "is it real?" with nothing installed, and returns the engine's answer as it wrote it:

import { secondOpinion } from "@oloproof/sdk";

const result = await secondOpinion({
  baseUrl: "https://oloproof.com",
  baseline: "scores/baseline.jsonl",
  candidate: "scores/candidate.jsonl",
});
if (result.ok) console.log(result.opinion.answers);
else console.log(result.error, result.description);

The page keeps nothing and needs no account. It reads up to 2 MB per file and is limited per client; a refusal returns the engine's sentence naming the row to change.

What is not built

  • Publishing @oloproof/sdk to npm. Until it is, build it from packages/sdk in the repository.
  • A statistic of any kind in Node, by design: the SDK writes scores and reads the engine's answer.
  • A group_id or time_order in your suite is not carried into the import, which reads case ids and scores only, so the engine reads the cases as independent.

Where to go next