# Launch an experiment - Obss

Run experiments
Launch an experiment
Choose the tasks, treatments, agents, and settings for an experiment, then launch it, launch it again, or cancel it.
Pick the tasks, treatments, and agents to compare, set the trials, and launch. A launch spends runs from your organization’s run balance. If you’re new to experiments, read
Experiments
first.
Prerequisites
At least one
task
. A task needs a rubric for its runs to get an Evals result.
At least one
treatment
. Every project has
Raw Agent
.
At least one
agent
with a model, under
Manage → Agents
.
Runs in your organization’s
run balance
.
Start a new experiment
Select
New experiment
in the sidebar and fill in
Experiment name
. You can also start from a
template
on
Home
, which fills in its tasks, treatments, and agents. Tasks already carry their files and machine, and treatments carry their tools, so in the
Configuration
panel you only choose what to combine.
Choose tasks
Select tasks in the task list. Search by name, or use the
Tags
filter to select all the tasks of a release or an area at once. To create a task without leaving the page, select
New task
.
Each task runs its current version. To run an older version, choose it in the task’s version menu.
To compare treatments or agents, select at least 6 tasks, and more if you can.
Comparisons
needs at least 6 tasks where one side did better, and tied tasks don’t count, however many trials you run.
Choose agents
A new experiment selects every built-in agent that has a recommended model. To pick your own, select
Clear
beside the agent search, then select the agents you want. You need at least one agent to launch. The number beside
Clear
is the number of agents, so an agent with two models counts once.
For each agent, choose a model and an effort level. To compare models, add the same agent again with another model.
A new experiment doesn’t select a
custom agent
for you unless you open
New experiment
from that agent’s page. A selected custom agent starts on
Agent default
, and one that is
Not tested
or
Needs attention
shows a notice before you launch.
Choose treatments and a baseline
Select the treatments to compare under
Treatments (Toolsets)
. A new experiment starts with
Raw Agent
selected.
When you select two or more treatments, a
Baseline
control appears. Choose the treatment you want to compare the others with. It can be any treatment you selected, and
Raw Agent
starts as the baseline when it’s selected.
Obss saves the baseline with the experiment, and you read each treatment’s
lift
by comparing it with the baseline in
Pass rate and lift
.
Set trials
Set
Trials
from 1 to 20. A new experiment starts at 1. Use two or more when you compare treatments or agents, because agents don’t give the same result every time.
The footer shows how many runs the experiment uses: tasks × treatments × agents × trials (see
What an experiment combines
). If your run balance doesn’t have enough, the footer links to where you can
add runs
.
Set Max Concurrency
Max Concurrency
limits how many runs are active at once. It starts at the number of runs, and it can’t be higher.
Lower it when runs change shared resources through connected MCP servers, CLIs, or other tools. A lower limit helps prevent race conditions, unexpected results, and rate-limit spikes at your model provider. Set it to 1 to run one at a time.
Max Concurrency doesn’t apply to Evals or Insights. Obss also applies organization and global limits, so the effective limit can be lower than the value you set. For help choosing a limit or for more capacity, email
support@obss.app
.
Turn on Evals and Insights
Control
What it does
Evals
Scores each run against its task rubric. When it’s off, runs aren’t scored.
Insights
Reads each run’s saved trace to find frictions, and finds patterns across the experiment.
Each control needs its models set under
Obss AI features
. Leave one off now and you can select
Run evals
or
Run insights
on the experiment later.
When
Evals
or
Insights
is on and your organization has Agent mode for it, choose its
Mode
:
Mode
How it reviews each run
Cost and time
Standard
Reviews each run in one pass with your Evals model or Insights model.
Fastest and lowest cost
Agent
Obss’s evals agent or insights agent reads the whole run and checks what it finds.
Costs more and takes a few minutes per run
With
Agent
chosen, the line under
Mode
says how many runs the agent will review, and with which model.
Agent mode for Evals and Insights is in private beta. Email
support@obss.app
to enable it for your organization. See
Agent mode
.
Override models for this experiment
Open
Advanced
to choose a different
Evals model
,
Insights model
, or
Experiment insights model
for this experiment only.
Default
uses your organization’s setting.
If your organization has
custom instructions
, turn on
Customize
under a feature to replace them for this experiment. Left off, the experiment uses the organization’s instructions, or Obss’s built-in guidance when there are none.
In a launch plan from the CLI or the plugin, the role overrides use these names. Omit a role to use the organization’s model.
Role
Name in the plan
Needs
Evals model
judge
Evals on
Insights model
sweep
Insights on
Experiment insights model
consolidation
Insights on
Launch
Dashboard
CLI
Plugin (MCP)
Check the run count in the footer and select
Launch
. If something blocks the launch, such as a tool that needs values or an agent without a provider, the panel names it and links to the fix.
Check the plan, launch it, and wait for it to finish:
obss
experiments
preflight
--plan-file
plan.json
obss
experiments
launch
--plan-file
plan.json
obss
experiments
watch
exp_8f2c41d0
The CLI launches as soon as you run
launch
, so run
preflight
first. A preflight changes nothing and lists each value a person must add, with its setup link.
The plan names the tasks, treatments, agents,
trialCount
,
maxParallelAgentSessions
,
evals
,
insights
,
evalsMode
, and
insightsMode
, and its
referenceTreatmentClientId
names the baseline. To write a plan, start from the
example plan
. Every flag is in
obss experiments launch
.
Ask your AI assistant to launch it. The assistant checks the plan with
Check an experiment plan before launch
(
preflight_experiment
), launches it with
Launch an experiment
(
launch_experiment
), and follows it with
Get experiment status
(
get_experiment_status
).
A launch spends runs, so turn on approval for Obss tools in your AI assistant’s tool approval settings, and approve each launch. Ask Obss always asks before a launch.
Launch an Obss experiment with the checkout tasks, Raw Agent and our Stripe MCP treatment, Claude Code and Codex, and 2 trials.
When you launch, Obss saves each task’s version, files, and machine, and each treatment’s tools, so later edits don’t change the runs. Results fill in on the experiment page as runs finish:
Matrix
shows each task against each agent and treatment for a metric such as Evals or cost,
Frictions
shows where agents got stuck, and
Runs
lists every run. Learn how to interpret them in
Read results
, or get a plain-language summary from
Ask Obss
.
Launch again
Launch again
starts a new experiment with the same name, tasks, treatments, agents, and settings, and uses as many runs as the original planned. It’s on the experiment and in its row menu in
All experiments
once the experiment is
Completed
,
Needs review
,
Error
, or
Cancelled
.
Latest versions:
each task runs its latest version, and each treatment uses its current saved state. If tasks changed since the experiment ran, the dialog says how many.
Blockers:
the dialog lists anything that blocks the launch, such as a disconnected provider, each with a fix link that opens in a new tab. When you come back, the dialog checks again, and
Launch
turns on as soon as nothing blocks it.
Deleted tasks or treatments:
Launch again
won’t launch, so it never runs a smaller experiment or compares against a different baseline. Use
Edit first
to choose others.
Edit first
opens
New experiment
with the same choices so you can change them. To launch again on a repeat, select
Schedule
.
Dashboard
CLI
Plugin (MCP)
Select
Launch again
, and then select
Launch
.
obss
experiments
rerun
exp_8f2c41d0
Pass
--versions experiment
to run the experiment’s own task versions instead of the latest. To change the plan first, save it with
obss experiments clone-plan
, edit it, and launch it. See
obss experiments rerun
.
Your AI assistant calls
Launch an experiment again
(
rerun_experiment
). To change the plan first, it reads it with
Get an experiment’s launch plan
(
get_experiment_plan
), edits it, and launches it.
The new experiment is a child of the one you launched it from, its parent experiment. In
All experiments
, its row shows
Parent:
and the parent’s name, and its
⋯
menu has
Open parent experiment
:
Cancel an experiment
To stop an experiment that is
Running
, select
Cancel experiment
on the experiment or in its row menu. It stops unfinished runs, and completed runs stay available. To stop one run, select
Cancel run
in that run’s
⋯
menu.
Dashboard
CLI
Plugin (MCP)
Select
Cancel experiment
, then confirm.
obss
experiments
cancel
exp_8f2c41d0
Add
--run-id
to cancel one run. The CLI asks you to confirm. See
obss experiments cancel
.
Your AI assistant calls
Cancel an experiment or run
(
cancel_experiment
). It asks you to approve the call.
From the CLI and the plugin
Task
CLI
Plugin (MCP)
Check a plan before launch
obss experiments preflight
preflight_experiment
Launch a plan
obss experiments launch
launch_experiment
Follow progress
obss experiments watch
,
obss experiments status
get_experiment_status
,
get_launch_status
Launch again
obss experiments rerun
rerun_experiment
Read a plan to edit
obss experiments clone-plan
get_experiment_plan
Cancel
obss experiments cancel
cancel_experiment
Next steps
Read results
Compare Evals results and cost in Matrix.
Schedules
Launch the experiment again on a repeat.
Frictions
See where agents got stuck.
obss experiments
Every flag for launch, watch, and cancel.
Experiments
Templates
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform