Skip to main content
Pick the tasks, treatments, and agents to compare, set the trials, and launch. A launch spends runs from your organization’s run balance. If you’re new to experiments, read Experiments first.

Prerequisites

  • At least one task. A task needs a rubric for its runs to get an Evals result.
  • At least one treatment. Every project has Raw Agent.
  • At least one agent with a model, under Manage → Agents.
  • Runs in your organization’s run balance.

Start a new experiment

Select New experiment in the sidebar and fill in Experiment name. You can also start from a template on Home, which fills in its tasks, treatments, and agents. Tasks already carry their files and machine, and treatments carry their tools, so in the Configuration panel you only choose what to combine.
New experiment page with the selected tasks and the Configuration panel: Agents, Treatments (Toolsets), Trials, Max Concurrency, Evals, and Insights.New experiment page with the selected tasks and the Configuration panel: Agents, Treatments (Toolsets), Trials, Max Concurrency, Evals, and Insights.

Choose tasks

Select tasks in the task list. Search by name, or use the Tags filter to select all the tasks of a release or an area at once. To create a task without leaving the page, select New task. Each task runs its current version. To run an older version, choose it in the task’s version menu. To compare treatments or agents, select at least 6 tasks, and more if you can. Comparisons needs at least 6 tasks where one side did better, and tied tasks don’t count, however many trials you run.

Choose agents

A new experiment selects every built-in agent that has a recommended model. To pick your own, select Clear beside the agent search, then select the agents you want. You need at least one agent to launch. The number beside Clear is the number of agents, so an agent with two models counts once. For each agent, choose a model and an effort level. To compare models, add the same agent again with another model. A new experiment doesn’t select a custom agent for you unless you open New experiment from that agent’s page. A selected custom agent starts on Agent default, and one that is Not tested or Needs attention shows a notice before you launch.

Choose treatments and a baseline

Select the treatments to compare under Treatments (Toolsets). A new experiment starts with Raw Agent selected. When you select two or more treatments, a Baseline control appears. Choose the treatment you want to compare the others with. It can be any treatment you selected, and Raw Agent starts as the baseline when it’s selected. Obss saves the baseline with the experiment, and you read each treatment’s lift by comparing it with the baseline in Pass rate and lift.

Set trials

Set Trials from 1 to 20. A new experiment starts at 1. Use two or more when you compare treatments or agents, because agents don’t give the same result every time. The footer shows how many runs the experiment uses: tasks × treatments × agents × trials (see What an experiment combines). If your run balance doesn’t have enough, the footer links to where you can add runs.

Set Max Concurrency

Max Concurrency limits how many runs are active at once. It starts at the number of runs, and it can’t be higher. Lower it when runs change shared resources through connected MCP servers, CLIs, or other tools. A lower limit helps prevent race conditions, unexpected results, and rate-limit spikes at your model provider. Set it to 1 to run one at a time. Max Concurrency doesn’t apply to Evals or Insights. Obss also applies organization and global limits, so the effective limit can be lower than the value you set. For help choosing a limit or for more capacity, email support@obss.app.

Turn on Evals and Insights

Each control needs its models set under Obss AI features. Leave one off now and you can select Run evals or Run insights on the experiment later. When Evals or Insights is on and your organization has Agent mode for it, choose its Mode: With Agent chosen, the line under Mode says how many runs the agent will review, and with which model.
Agent mode for Evals and Insights is in private beta. Email support@obss.app to enable it for your organization. See Agent mode.

Override models for this experiment

Open Advanced to choose a different Evals model, Insights model, or Experiment insights model for this experiment only. Default uses your organization’s setting. If your organization has custom instructions, turn on Customize under a feature to replace them for this experiment. Left off, the experiment uses the organization’s instructions, or Obss’s built-in guidance when there are none. In a launch plan from the CLI or the plugin, the role overrides use these names. Omit a role to use the organization’s model.

Launch

Check the run count in the footer and select Launch. If something blocks the launch, such as a tool that needs values or an agent without a provider, the panel names it and links to the fix.
When you launch, Obss saves each task’s version, files, and machine, and each treatment’s tools, so later edits don’t change the runs. Results fill in on the experiment page as runs finish: Matrix shows each task against each agent and treatment for a metric such as Evals or cost, Frictions shows where agents got stuck, and Runs lists every run. Learn how to interpret them in Read results, or get a plain-language summary from Ask Obss.

Launch again

Launch again starts a new experiment with the same name, tasks, treatments, agents, and settings, and uses as many runs as the original planned. It’s on the experiment and in its row menu in All experiments once the experiment is Completed, Needs review, Error, or Cancelled.
  • Latest versions: each task runs its latest version, and each treatment uses its current saved state. If tasks changed since the experiment ran, the dialog says how many.
  • Blockers: the dialog lists anything that blocks the launch, such as a disconnected provider, each with a fix link that opens in a new tab. When you come back, the dialog checks again, and Launch turns on as soon as nothing blocks it.
  • Deleted tasks or treatments: Launch again won’t launch, so it never runs a smaller experiment or compares against a different baseline. Use Edit first to choose others.
Edit first opens New experiment with the same choices so you can change them. To launch again on a repeat, select Schedule.
Select Launch again, and then select Launch.
The new experiment is a child of the one you launched it from, its parent experiment. In All experiments, its row shows Parent: and the parent’s name, and its ⋯ menu has Open parent experiment:
All experiments with two rows named Run again: Stripe CLI and Raw Agent, each showing Parent: Stripe CLI and Raw Agent, 7 create tasks. The first row's menu is open, with Open parent experiment at the top.All experiments with two rows named Run again: Stripe CLI and Raw Agent, each showing Parent: Stripe CLI and Raw Agent, 7 create tasks. The first row's menu is open, with Open parent experiment at the top.

Cancel an experiment

To stop an experiment that is Running, select Cancel experiment on the experiment or in its row menu. It stops unfinished runs, and completed runs stay available. To stop one run, select Cancel run in that run’s ⋯ menu.
Select Cancel experiment, then confirm.

From the CLI and the plugin

Next steps