# Agent mode - Obss

Evals and Insights
Agent mode
Have an Obss agent check each run in depth for Evals or Insights, see how many runs it will review, and read what it found.
Agent mode for Evals and Insights is in private beta. Email
support@obss.app
to enable it for your organization.
In
Agent mode
, Obss runs an agent on your own Evals model or Insights model. The agent opens the whole run and calls the model many times before it answers, instead of the one call of Standard mode.
For
Evals
, Obss’s evals agent also reads the changed files, and checks the agent’s claims before it scores.
For
Insights
, Obss’s insights agent checks each friction against what the tools returned. On
Comparisons
, it also shows
what the runs did differently
for each difference.
Evals and Insights each have their own mode, so you can use Agent mode for one and Standard mode for the other. Your organization gets Agent mode for each one separately, so when you email, say whether you want it for Evals, Insights, or both.
Standard and Agent modes
Both modes use the same models and give the same kind of result: Evals score against the same rubric, and Insights report frictions in the same way.
Standard
Agent
How it reviews a run
One pass. Long runs are condensed to fit the model’s context window.
Reads the whole run and checks what it finds
Quality
One read of what fits
Finds problems that one pass can miss
Cost
Lowest
Higher: many calls to your models
Time
Fastest
A few minutes per run
Use it when
Runs are short, or speed and cost matter most
Runs are long or complex, a score depends on checking the work, or one pass misses frictions
Agent mode doesn’t change the runs an experiment uses. Your model provider bills the extra model calls directly, as for all
model use
.
Because Agent mode doesn’t condense the run, a note about what it read shows only when the saved run is incomplete, for example when the trace was too large to save (
What Evals read
,
What Insights read
).
Choose models for Agent mode
Agent mode needs models that can call tools and have a known context window: the
Evals model
for Evals, and both the
Insights model
and the
Experiment insights model
for Insights. If a model can’t run Agent mode, you can’t select
Agent
, and the reason shows under it. The CLI and the plugin refuse the request with
agent_mode_model_unsupported
and a link to
Obss AI features
. Choose another model under
Obss AI features
, or use Standard mode.
Agent mode can’t use Claude Pro / Max models. With Claude Pro / Max, choose a model from an API-key provider, or use Standard mode.
Turn on Agent mode
Choose the mode at launch, or when you run Evals or Insights again. If
Mode
doesn’t show, your organization doesn’t have Agent mode for that feature yet.
Dashboard
CLI
Plugin (MCP)
In the new experiment’s
Configuration
panel, turn on
Evals
,
Insights
, or both.
Under each one, set
Mode
to
Agent
.
Mode
shows only while its feature is on.
Run evals
,
Rerun evals
,
Run insights
, and
Rerun Insights
offer the same choice. Each one starts on the mode of the latest attempt that ran. If no attempt ran, it starts on the experiment’s launch mode.
obss
evals
trigger
exp_8f2c41d0
--mode
agent
obss
insights
trigger
exp_8f2c41d0
--mode
agent
Without
--mode
, an attempt uses the experiment’s launch mode, not the latest attempt’s mode as the dashboard does. To repeat the latest mode, pass the preview’s
mode.latestMode
. In a launch plan, set
"evalsMode": "agent"
with
"evals": true
, or
"insightsMode": "agent"
with
"insights": true
.
obss evals trigger
and
obss insights trigger
list every flag.
Ask your AI assistant to run Evals or Insights in Agent mode. It calls
Start Evals for an experiment
(
trigger_evals
) or
Start Insights for an experiment
(
trigger_insights
) with
mode: "agent"
, and you approve each call. Without
mode
, an attempt uses the experiment’s launch mode, not the latest attempt’s mode as the dashboard does.
Here is the
Configuration
panel with
Mode
set to
Agent
for both Evals and Insights:
Check what Agent mode uses
Before you start, the line under
Mode
says how many runs the agent will review, and with which model, for example “Up to 56 agent reviews on Claude Opus 5.5.” When the Insights model and the Experiment insights model are different, the line names both: ”… on Claude Sonnet 5.5 (Insights model) and Claude Opus 5.5 (Experiment insights model).” For Insights, the count covers only the run reviews. Grouping the frictions and reading the pairs of runs for
What the runs did differently
add to it.
Here is the
Rerun Insights
dialog with
Agent
chosen, above the two models that the insights agent uses:
As the agent works, Obss adds up what it used. When the experiment has more than one attempt, the attempt menu on
Matrix
shows it, for example “Agent mode · 2.4M input tokens on Claude Opus 5.5 (300K cached)”. The input count includes every token the model read, and the cached part is what your provider read from its prompt cache. For Insights, the count also includes grouping the frictions and reading the pairs of runs. It names a model only when the Insights model and the Experiment insights model are the same.
From the CLI and the plugin:
Before a launch,
obss experiments preflight --output json
and
preflight_experiment
return
agentModeEstimates
.
Before a rerun,
obss evals preview
and
obss insights preview
print the same line, and
preview_evals
and
preview_insights
return it as
mode.estimate
.
For every attempt,
obss evals list
,
obss insights list
,
list_evals
, and
list_insights
return what the agent used as
agentUsage
.
Read Evals in Agent mode
A run scored in Agent mode shows
Agent mode
next to its
Evals model
on the
Evals
tab. The info icon next to the experiment status has an
Evals mode
row with the value
Agent
, and when there is more than one Evals attempt, the attempt menu on
Matrix
shows
Agent mode
. Standard mode has no label, and the CLI and the plugin report each attempt’s mode.
Each criterion shows the evals agent’s explanation and its citations: run steps, diff lines, and changed files. Select a file citation to open it on
Output
. Here, a criterion fails because the agent’s explanation of the bug doesn’t match the code it changed.
During the review, the
Evals
tab says “Obss’s evals agent is reviewing this run. Results can take a few minutes.” The CLI and the plugin report the run as running.
Read Insights in Agent mode
A run’s trace shows
Agent mode
next to its
Insights model
, and the info icon next to the experiment status has an
Insights mode
row with the value
Agent
. When there is more than one Insights attempt, the attempt menu on
Matrix
shows
Agent mode
too. While the agent reviews a run, the run’s trace says “Obss’s insights agent is reviewing this run. Its frictions can take a few minutes to appear.”
When it reviews a run, the insights agent doesn’t read the changed files, the rubric, or the Evals verdicts, as in Standard mode.
What the runs did differently
In Agent mode, each difference’s block on
Comparisons
that has a Brief ends with
What the runs did differently
. Say Raw Agent did worse than Stripe CLI. For each task where Raw Agent did worse, the insights agent reads a pair of runs: a failed Raw Agent run next to a passed Stripe CLI run. For each run, it writes one line about the step where the two runs differ, and quotes up to five lines of that step, linked to it in the run’s trace.
When the agent has read every pair, the
Brief
sentence says once what the runs did across the tasks, for example “Raw Agent runs found no Stripe CLI or Stripe key, while Stripe CLI runs found the CLI and a test key.” The section then shows one example task with “Like 6 other tasks” under its name. A task that went differently from the rest shows next, with “Went differently” under its name, and the Brief sentence names it.
A row shows one failed run and one passed run of its task. It shows where these two runs went different ways, not why one side did worse, and the task’s other runs can differ.
Show all N tasks
opens every row.
Before the agent has read every pair, the first three rows show, and a line says how many it is still reading, for example “Reading 12 more tasks…”. The Brief is ready before the pairs are, and Obss writes its sentence again when the last pair is read.
When a side’s repeated runs disagree, a line under the task says so, for example “Raw Agent failed 2 of 3 runs”.
Each row’s
⋯
menu has
Open both runs
. If the agent couldn’t explain a pair, the row links the two runs instead, and the next Insights attempt tries that pair again.
Copy as issue
includes the rows that the section shows first.
Only runs that Evals marked Pass or Fail are paired, and for a pair the insights agent knows only which run failed and which passed.
When Agent mode fails
If an agent stops before it returns its result, the result says why and what to do:
For Evals, the run gets no
Pass
or
Fail
, and its
Evals
tab says why.
For Insights, a run the agent couldn’t review has no frictions from that attempt.
obss insights get
and
get_insights_results
say why for each run.
If the insights agent couldn’t group the experiment’s frictions, each run’s review still stands. The Insights attempt’s
error
in
obss insights list
and
list_insights
says why.
For causes and fixes, see
Agent mode did not return a result
.
Next steps
Comparisons
See which differences hold up, and what the runs did differently.
Frictions
Rank the problems that Insights found, and decide what to fix.
Obss AI features
Choose the models that Agent mode uses.
obss evals and insights
Every command and flag.
Evals
Insights
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform