# Oloproof > Your eval score went up. Is it real? And can you trust the judge that measured it? Oloproof puts an interval behind every decision and decides one of four states. These docs describe Oloproof 0.1.0a16 (`pip install oloproof==0.1.0a16`). The four decision rules, in plain words: - PASS: the whole confidence interval clears the bar you set (for a minimum threshold T, the lower bound L >= T). - FAIL: the whole confidence interval misses the bar (the upper bound U < T). - INSUFFICIENT_EVIDENCE: the interval straddles the bar. Not established yet, and here is what it would take: Oloproof says how many more cases would settle it. - MANUAL_REVIEW: a person is asked: the policy requires it for this rule, an evaluator behind it is retired, or the method declined to answer. A run with no bar decides nothing. First action: `pip install oloproof`, then `oloproof scores import before.csv after.csv`. Every page below as one file: https://oloproof.com/llms-full.txt ## What Oloproof does - [Start here](https://oloproof.com/docs/start-here.md): What Oloproof does, when to use it and when not to, and the order to read these guides in. - [Core concepts](https://oloproof.com/docs/concepts.md): Oloproof tells you whether your AI system is good enough, with the uncertainty attached, before it ships and once it is doing real work. - [What works today](https://oloproof.com/docs/what-works.md): What you can evaluate with Oloproof today, how you reach it (Python SDK, oloproof.yaml and the CLI, or the browser), and what is not built. - [What an evaluation cannot tell you](https://oloproof.com/docs/limits.md): Every Oloproof result is a measured claim with stated limits. ## Your first evaluation - [Quickstart](https://oloproof.com/docs/quickstart.md): Install the engine, scaffold a project, and get a first run and a gate decision on your own machine. - [From zero to a verdict on your own app](https://oloproof.com/docs/your-own-app.md): You have a Python function that answers questions, and you want to know if it is good enough to ship. ## Guides by system type - [Tutorial: a classifier or structured output](https://oloproof.com/docs/tutorial-classification.md): Evaluate a Python function that labels support questions, read why the release is blocked, fix the misses, and compare the fix with the original, all on your own machine with no account, no network and no model. - [Tutorial: text generation with a rubric judge](https://oloproof.com/docs/tutorial-generation.md): Evaluate a function that writes free text, here a ticket summariser, with format checks and a rubric judge; measure that judge against a person's labels before it may decide anything; then compare a real change. - [Tutorial: an application behind an HTTP endpoint](https://oloproof.com/docs/tutorial-http.md): Evaluate a service you reach over HTTP, without importing its code: point Oloproof at the URL, run the suite, find what it misses, deploy a change, and compare. - [Tutorial: evaluate a RAG application](https://oloproof.com/docs/tutorial-rag.md): A runnable walkthrough for a retrieval-augmented application, in two paths: an existing black-box application whose retrieval, context and citations you record from the outside, and a staged application Oloproof runs stage by stage so oloproof diagnose can re-execute failed cases under controlled changes. - [Tutorial: evaluate an agent](https://oloproof.com/docs/tutorial-agents.md): A runnable walkthrough for a tool-using agent and for a team of agents: record what the agent did as a trajectory, check its tool use, constraints, steps, routing, permissions and hand-offs, gate a release on them, and compare a candidate change with the baseline. - [Tutorial: a classifier and a regressor](https://oloproof.com/docs/tutorial-predictive.md): A runnable walkthrough for three predictive models evaluated from the command line: a binary churn classifier, a three-class ticket router evaluated one class at a time, and a delivery-time regressor scored by absolute error within a declared range. - [Tutorial: a multi-turn conversation](https://oloproof.com/docs/tutorial-conversation.md): A runnable walkthrough for evaluating a conversational assistant with the Python SDK: your application replays a scripted conversation against a fresh session, records what it answered as a conversation/v1 artifact, and two evaluators judge each whole conversation. - [Tutorial: evaluate a reinforcement-learning policy](https://oloproof.com/docs/tutorial-rl.md): A runnable walkthrough for two kinds of reinforcement-learning question: how well a policy you can run performs over episodes, and how well a policy you cannot try in the world would have done, judged from the log of the policy it would replace, one decision at a time or several per episode. - [Tutorial: images, with checks and a judge that sees them](https://oloproof.com/docs/tutorial-media.md): Evaluate a function that makes images, here a product thumbnailer that also writes a caption. - [Tutorial: a support desk against simulated customers](https://oloproof.com/docs/tutorial-simulated-users.md): Evaluate a conversational assistant against users who react to what it says. ## Inspect and understand results - [Writing a suite](https://oloproof.com/docs/suites.md): A suite is two files and a dataset. - [Errors](https://oloproof.com/docs/errors.md): A run that failed and a run that errored are different things. - [Clustered cases](https://oloproof.com/docs/clusters.md): Most interval arithmetic assumes every case is independent of every other. - [Slices](https://oloproof.com/docs/slices.md): A global rate can hold steady while one part of the suite collapses. - [RAG evaluation](https://oloproof.com/docs/rag.md): A retrieval-augmented answer can be wrong for four different reasons, and a single accuracy number cannot tell them apart. - [Agents and tools](https://oloproof.com/docs/agents.md): Oloproof does not drive an agent. - [Classifiers and regressors](https://oloproof.com/docs/predictive.md): A predictive model is evaluated like any other system: a callable returns a prediction for each case, and evaluators read it. - [Reinforcement learning](https://oloproof.com/docs/rl.md): Oloproof evaluates a reinforcement-learning policy by running it in episodes, or from the log of the policy it would replace. - [Images, audio and video](https://oloproof.com/docs/media.md): Oloproof evaluates systems that take or produce images, audio and video, and the files never leave the place your system keeps them. - [Simulated users](https://oloproof.com/docs/simulated-users.md): A scripted conversation tests your assistant against a user who never reacts. - [The Python API](https://oloproof.com/docs/sdk.md): Everything the CLI does, the library does. - [Evaluating from Node](https://oloproof.com/docs/node-sdk.md): The Node SDK runs your system and its evaluators in Node, inside your own test runner, and hands the scores to the Python engine, which decides. - [Recording what a system did](https://oloproof.com/docs/tracing.md): A system's output is what an evaluator reads by default. - [Progress and concurrency](https://oloproof.com/docs/streaming.md): A suite against a live model takes minutes, and most of them are spent waiting on the model. ## Compare versions - [Comparing a candidate to a baseline](https://oloproof.com/docs/compare.md): One of the two ways in, beside holding one system to its requirement: you changed something, and you want to know whether it helped. - [Comparison rules](https://oloproof.com/docs/comparison-rules.md): Every kind of comparison rule: what each asks, what a margin is measured in, and what moves the interval it is read from. - [A second opinion on another tool's scores](https://oloproof.com/docs/second-opinion.md): Your eval score went up. ## Review, collaboration and CI - [Judges](https://oloproof.com/docs/judges.md): An LLM judge is one kind of evaluator, not all of them. - [Gating CI](https://oloproof.com/docs/gating.md): A gate compares measured evidence against a threshold you declared, and exits with a code your CI understands. - [Risk ceilings](https://oloproof.com/docs/risk-ceilings.md): A ceiling bounds how often, or how much, something goes wrong: unsafe answers, leaked data, attacks that succeed, a cost or a latency. - [Hazard register](https://oloproof.com/docs/hazards.md): A hazard is something that can go wrong with your system, written down with who owns it and what measures it. - [Generating attacks](https://oloproof.com/docs/attack-generation.md): Oloproof writes adversarial test cases for a system you own and run in a sandbox, and checks what the system does with them. - [Formal proofs](https://oloproof.com/docs/proofs.md): A proof says something an evaluation cannot: that a property holds in every case of a model, not only in the cases a suite holds. - [Assurance cases](https://oloproof.com/docs/assurance-cases.md): An assurance case is your team's argument that a system is good enough for a stated use: each claim, the use it is made for, and the evidence it rests on. - [Evidence graph](https://oloproof.com/docs/evidence-graph.md): The evidence graph is everything a release decision rests on, written as typed nodes and typed edges in one JSON file. - [NIST AI RMF and OSFI E-23](https://oloproof.com/docs/frameworks.md): What evidence Oloproof records toward the NIST AI RMF's Measure function and OSFI's Guideline E-23, item by item, and what it does not record. - [Running in CI](https://oloproof.com/docs/ci.md): Running Oloproof inside a CI job, and comparing a pull request against the branch it targets. - [Managed runs](https://oloproof.com/docs/managed-runs.md): A managed run executes your suite on Oloproof's workers instead of your machine, and you watch it from the Jobs page or your terminal. - [Notifications](https://oloproof.com/docs/notifications.md): A passing run notifies nobody. ## Reference - [Configuration reference](https://oloproof.com/docs/reference-config.md): Every field of oloproof.yaml and release.yaml, with its type, its default, the values it accepts and an example, taken from the models that read the files. - [SDK reference](https://oloproof.com/docs/reference-sdk.md): Every name the oloproof and oloproof.evaluators packages export, with its signature, whether it is synchronous or asynchronous, and what it returns. - [Results and execution reference](https://oloproof.com/docs/reference-results.md): The exact contracts behind a run, from the HTTP exchange to the exit codes. - [CLI reference](https://oloproof.com/docs/cli.md): Every command Oloproof registers, with what it does.