Skip to main content
Run an experiment to find out which tools and agents do your work best. It runs every combination of the tasks, treatments, and agents you choose, from the same starting point.

What an experiment combines

Each combination is one run: one agent, with one treatment, doing one task once. Files and a machine attach to a task, tools attach to a treatment, and you choose agents at launch. Learn more in Where files, machines, and tools attach. For example, 3 tasks × 2 treatments × 2 agents × 2 trials makes 24 runs, which uses 24 from your organization’s run balance.

Why the comparison is fair

An experiment changes only what you compare. Two treatments on the same task differ only in their tools, and two agents with the same treatment differ only in agent and model.
  • Same starting point: every run of a task starts from that task’s files on that task’s machine.
  • Fresh machine: each run gets a new machine, so nothing carries over between runs.
  • Snapshot at launch: Obss saves the task version, its files and machine, and each treatment’s tools at launch, so later edits don’t change launched runs.

Baseline and lift

The baseline is the treatment you compare the others with. It can be any treatment in the experiment. Most experiments use Raw Agent, which every project has: it runs the agent as its vendor ships it, with no added tools. Lift is the difference in pass rate between a treatment and the baseline, on the same tasks and agents. For example, if agents pass 9 of 20 runs with Raw Agent (45%) and 15 of 20 with your MCP server’s treatment (75%), the lift is 30 percentage points. A small lift doesn’t show that the treatment helped: Comparisons tells you whether it holds up. When you select two or more treatments, you choose the Baseline at launch. Read lift in Pass rate and lift.

Trials

Agents don’t give the same result every time, so a trial repeats the same inputs. Each trial is a separate run. Start with two trials. More trials make each pass rate steadier, but they don’t replace tasks: to show that two pass rates really differ, Comparisons needs at least 6 tasks where one side did better. Tied tasks don’t count, so write more than 6.

Experiment statuses

The status shows on All experiments and on the experiment page, and the CLI and the plugin return its Value. Right after you launch, the experiment page can show Starting your experiment for a few seconds while Obss prepares the runs. After a launch from the CLI or the plugin, get_launch_status returns admitting until the experiment exists, or admission_failed if the launch didn’t start.

What you can do with an experiment

You can start an experiment from New experiment, a template, Launch again, the CLI, or your AI assistant. Every route makes the same kind of experiment, and the results appear in the same place.

Next steps