Skip to main content
Agents don’t pass the same task every time, so a small gap in pass rate may not hold up. Open an experiment and select the Comparisons tab to see which differences do. It compares Evals verdicts, so it needs Evals.
A difference needs at least 6 tasks where one side did better than the other. With fewer tasks, no difference can show, however many runs you add.

Answers

Comparisons opens with one sentence that answers your experiment’s question, and then one block for each difference:
Stripe CLI did better than Raw Agent on all 7 tasks, then its block: 93% and 0%, Clear difference, then the Brief and two frictions.Stripe CLI did better than Raw Agent on all 7 tasks, then its block: 93% and 0%, Clear difference, then the Brief and two frictions.
The sentence covers your experiment’s main comparisons: each treatment against the baseline, for all agents together. With one agent, they’re that agent’s comparisons. With one treatment, they’re each pair of agents. For example:
  • “Stripe CLI did better than Raw Agent on all 7 tasks.”
  • “Stripe MCP likely did better than Raw Agent on 6 tasks, same on 4.” A likely result is never stated as settled.
  • “No treatment did clearly better than All official CLIs across all agents.”
A line under the sentence counts the likely results for one agent alone or between agents, for example “7 other comparisons show a likely difference. See All 40 comparisons.” Each difference then gets a block of its own, and every block has the same parts: the two sides with their pass rates, how the tasks went (“Better on 14 tasks, worse on 2, same on 26.”), the result and its next step, and how each agent did, with See each agent to open those rows in All comparisons. A main comparison gets a block when its result is a Clear difference or a Likely difference. A result for one agent alone or between agents gets one only when it’s a Clear difference. A comparison with no clear difference gets no block: the sentence says it, and All comparisons lists it. Up to three blocks show, and Show all N differences opens the rest. Comparisons read the latest Evals attempt. To read an earlier one, choose it in the attempt menu, as on the Matrix. The Matrix also shows the sentence in one line above its grid.

Brief

When Insights has reviewed the runs, each difference’s block gets a Brief, up to 5 per experiment. Its sentence says what the side that did worse ran into, for example “Raw Agent runs had no authenticated Stripe access.” Under it, “Frictions on the 7 tasks where Raw Agent did worse” lists up to 3 frictions from that side’s runs, with how many of that side’s failed runs and the other side’s passed runs had each one (“Raw Agent 24 of 28 failed runs”, “Stripe CLI 0 of 26 passed runs”). Select a friction to open its suggested fix. A friction under a result was seen with the side that did worse. That doesn’t mean it caused the difference.
  • Copy as issue: a block with a Brief has a ⋯ menu. Copy as issue copies the difference as Markdown for GitHub or another tracker: its sides, result, sentence, frictions, and links, and in Agent mode the rows of What the runs did differently that the block shows first.
  • After a fix: when a Clear difference lists a friction to fix in your product, After a fix, run the N tasks that differed again · N runs under the list opens a new experiment with those runs, so you can check your fix.
  • What the runs did differently: with Insights in Agent mode (private beta), a block with a Brief ends with pairs of runs: on each task where that side did worse, one of its failed runs next to a passed run of the other side, and the step where they differ. Once every pair is read, the sentence says what the runs did across the tasks, and the section shows one example task and any task that went differently.
When nothing differs, there’s no Brief. The experiment’s frictions are on its Frictions tab.
  • Which frictions: a friction is listed when it’s in at least 3 runs of the side that did worse, on at least 2 tasks, and in a larger share of that side’s runs than the other side’s. The largest gap comes first. Runs that Insights didn’t review are left out, and a line under the list says how many. Frictions caused by an Obss error are left out.
  • Numbers: Obss picks the differences and counts the runs. No model picks or counts anything.
  • The sentence: your organization’s Experiment insights model writes it, with the model access of the person who started the Evals or Insights, and without your custom instructions. See Obss AI features. Obss rejects a sentence with a number or a claim of cause, and asks again. With Insights in Agent mode, once the insights agent has read every pair of runs of a difference, the sentence also reads their lines and says what the runs did across the tasks.
  • Updates: Obss writes the Brief when the experiment, its Evals, and its Insights finish, and again each time Evals or Insights finishes. In Agent mode, it also writes the Brief again once, when the last pair of runs is read. Until a new sentence is ready, the old one is hidden. Only the latest Evals attempt shows a Brief.

Result labels

The result says how sure the answer is: Point to a result to see what it means and its p-values. How results are decided explains the rules.

All comparisons

Under the blocks, All N comparisons lists every comparison in one table:
All 6 comparisons: Stripe CLI and Raw Agent across all agents is a Clear difference. With each agent on its own, it's a Likely difference with Run again to be sure · 42 runs. The two agents show no clear difference with either treatment.All 6 comparisons: Stripe CLI and Raw Agent across all agents is a Clear difference. With each agent on its own, it's a Likely difference with Run again to be sure · 42 runs. The two agents show no clear difference with either treatment.
The rows are in groups: With more than 6 agents, Obss doesn’t compare agents with each other.

Run a comparison again to be sure

You can run a Likely difference again: Obss plans a new experiment with the fewest runs needed to settle it. Launching it uses runs from your run balance. Run again to be sure · N runs appears once the experiment and its Evals finish, on the latest Evals attempt. A Likely difference shows Needs more tasks to be sure instead when fewer than 6 of its tasks can still show a difference, or when settling it would take more than 10 runs of each task.
On the block’s result line, or on its row in All N comparisons, select Run again to be sure · N runs. New experiment opens with those runs selected. Check the run count in the footer, then select Launch.If the comparison can no longer be run again, for example because Evals ran again or a treatment was deleted, Comparisons reopens with a note that explains why.
When the new experiment finishes, open its Comparisons. It has only one main comparison, so if the result holds, it’s a Clear difference. Whether a treatment helps one agent more is the exception: in the new experiment it’s still a breakdown, so it can come out likely again.
  • Tasks: every task from the comparison that can still show a difference, with at least 6. A task is left out when it already ran at least 4 times on both sides together, and every run passed or every run failed.
  • Sides: the comparison’s treatments and agents. For all agents together, every agent.
  • Runs each: the fewest that are likely to settle the difference, from the experiment’s own (at least 3) up to 10.
  • Versions: the experiment’s task versions, and each treatment’s current tools.
  • Evals: on, with the mode, Evals model, and instructions of the Evals attempt that found the difference.
  • Name: “Run again:”, followed by the comparison.

Empty states

When there’s nothing to compare, the tab shows one of these messages: Public experiment links don’t show Comparisons.

How results are decided

On the dashboard, open How results are decided under the table. It sums up the rules below and lists what applies to your experiment:
How results are decided, open: five lines on how two sides are compared and what each result means, then what applies to this experiment, such as the main comparisons and breakdowns, and that with 7 tasks a result for one agent alone can be a likely difference at most.How results are decided, open: five lines on how two sides are compared and what each result means, then what applies to this experiment, such as the main comparisons and breakdowns, and that with 7 tasks a result for one agent alone can be a likely difference at most.
Obss compares tasks, not runs. For each comparison, it counts the tasks where one side passed more often than the other, and ignores the tasks where both did the same. An exact two-sided sign test gives the p-value: the probability of a split at least this uneven if neither side were really better. Five tasks that all go one way give p = 0.063, and six give p = 0.031, which is why a difference needs at least 6 tasks. To keep each task fair, both sides use the same number of runs with a verdict, and every agent counts equally. So Runs passed can be lower than the run count on the Matrix.

Main comparisons and breakdowns

The more comparisons an experiment makes, the more likely one of them shows a large gap that isn’t real. So Obss checks the comparisons in two groups, and applies the Holm correction within each group:
  • Clear difference: p-value below 0.05 after the correction within its group.
  • Likely difference: below 0.05 before the correction, but not after.
  • No clear difference: everything else.
For example, an experiment with 2 treatments, 2 agents, and 7 tasks has 1 main comparison and 5 breakdowns. Stripe CLI did better on all 7 tasks, for both agents: p = 0.016. For all agents together, that’s a Clear difference. For each agent on its own, it’s a Likely difference, because it’s checked together with 4 other breakdowns. How results are decided then says “With 7 tasks, a result for one agent alone or between agents can be a likely difference at most.” When there’s only one main comparison, nothing corrects it, so it’s never a Likely difference: it’s clear or not. A breakdown can be clear when the main comparison isn’t, for example when a treatment helped one agent and hurt another. How results are decided also lists what limits these results, such as tasks that passed in every run, runs with no Evals verdict, or how often repeated runs disagreed. While the experiment or its Evals is still running, a line at the top of the page says the results can still change.

From the CLI and the plugin

Each command gives the sentence first, then each difference, in the dashboard’s words. Its reference page lists every field.

Next steps