Skip to main content
Evals tell you whether each finished run met the task’s rubric. Your Evals model marks each criterion pass or fail with an explanation, and those verdicts become the pass rates on the Matrix. Evals judge the outcome, while Insights look at how the agent worked and find frictions. You can turn on either, both, or neither, and rerun each one on its own.

Turn on Evals

1

Connect a model provider

Connect a model provider under Organization settings → Model providers.
2

Assign the Evals model

Assign the Evals model under Organization settings → Obss AI features.
3

Choose Evals for an experiment

Turn on Evals when you launch an experiment, or choose Run evals on an experiment that already ran.

Read the verdicts

Open a run and select Evals to see whether it passed, how many criteria it met, and which Evals model scored it. Expand a criterion to read its explanation.
Run Evals tab with a pass or fail verdict for each rubric criterion and the Evals model that scored it.Run Evals tab with a pass or fail verdict for each rubric criterion and the Evals model that scored it.
Because a model judges the run, write each criterion so that a reader could mark it true or false for one run.

Choose a mode

If your organization has Agent mode for Evals, pick a mode when you launch, or in Run evals or Rerun evals. Both use your Evals model and the same rubric.

What Evals read

Your Evals model reads the rubric, the run’s recorded steps, and the changed files. If the run doesn’t fit the model’s context window, a note on the Evals tab (and in the Evals column of Compare) says what was left out:
  • A numbered range of steps didn’t fit. The model gets a count of those steps by kind and tool instead.
  • A long step was shortened.
  • Changed files were missing, previewed, shortened, or left out.
  • The trace is larger than the 32 MB limit for one run file, so its steps couldn’t be read. Obss doesn’t save a trace that large, and the run keeps its outcome.
No note means the model read everything. A verdict covers only what the model read. If the work still doesn’t fit after Obss shortens it, Evals fail for that run and the message names the Evals model. Choose a model with a larger context window under Obss AI features and run Evals again. A failed Evals read isn’t a friction.
Runs processed before Obss added these notes show none, so on an older run a missing note doesn’t prove a full read.

Which models ran

Each Evals attempt keeps the model it started with, even if you change the Evals model later. Models show as provider, model, and effort, for example Evals model: OpenAI · GPT-5.4 · high, where default effort is the model’s own default. An attempt that didn’t run shows no model. An older attempt from before Obss recorded models shows Not recorded in the attempt menu, the experiment details, and the CLI.

Pending verdicts

Evals read each run as it finishes. Rerun evals clears the verdicts it will rewrite, and the run’s Evals tab says Evals pending until the new one arrives. When Evals won’t give a verdict for an attempt, Obss says so. From the CLI and the plugin, obss evals get says whether each run’s verdict is waiting, final, or won’t come. obss results analyze counts runs still waiting separately from inconclusive runs.

Run Evals again

Rerunning Evals scores the same recorded work without running the agent again. If the rubric was wrong, fix it on the task, then rerun Evals. If the task itself changed, launch a new experiment instead. When Evals finish, Obss writes the experiment’s Brief again if it has Insights.
Open the experiment and choose Rerun evals (or Run evals if Evals haven’t run yet). You can change the Evals model and the mode for this Evals attempt.

From the CLI and the plugin

Next steps

Agent mode

Have Obss’s evals agent review each run in depth.

Insights

Find out where agents got stuck.

Read results

Read pass rates in the Matrix, and compare them for lift.

obss evals and insights

Every command and flag.