What an experiment combines
Each combination is one run: one agent, with one treatment, doing one task once.
Files and a machine attach to a task, tools attach to a treatment, and you choose agents at launch. Learn more in Where files, machines, and tools attach.
For example, 3 tasks × 2 treatments × 2 agents × 2 trials makes 24 runs, which uses 24 from your organization’s run balance.
Why the comparison is fair
An experiment changes only what you compare. Two treatments on the same task differ only in their tools, and two agents with the same treatment differ only in agent and model.- Same starting point: every run of a task starts from that task’s files on that task’s machine.
- Fresh machine: each run gets a new machine, so nothing carries over between runs.
- Snapshot at launch: Obss saves the task version, its files and machine, and each treatment’s tools at launch, so later edits don’t change launched runs.
Baseline and lift
The baseline is the treatment you compare the others with. It can be any treatment in the experiment. Most experiments use Raw Agent, which every project has: it runs the agent as its vendor ships it, with no added tools. Lift is the difference in pass rate between a treatment and the baseline, on the same tasks and agents. For example, if agents pass 9 of 20 runs with Raw Agent (45%) and 15 of 20 with your MCP server’s treatment (75%), the lift is 30 percentage points. A small lift doesn’t show that the treatment helped: Comparisons tells you whether it holds up. When you select two or more treatments, you choose the Baseline at launch. Read lift in Pass rate and lift.Trials
Agents don’t give the same result every time, so a trial repeats the same inputs. Each trial is a separate run. Start with two trials. More trials make each pass rate steadier, but they don’t replace tasks: to show that two pass rates really differ, Comparisons needs at least 6 tasks where one side did better. Tied tasks don’t count, so write more than 6.Experiment statuses
The status shows on All experiments and on the experiment page, and the CLI and the plugin return its Value.
Right after you launch, the experiment page can show Starting your experiment for a few seconds while Obss prepares the runs. After a launch from the CLI or the plugin,
get_launch_status returns admitting until the experiment exists, or admission_failed if the launch didn’t start.
What you can do with an experiment
You can start an experiment from New experiment, a template, Launch again, the CLI, or your AI assistant. Every route makes the same kind of experiment, and the results appear in the same place.
