Skip to main content
Results can shift when your product, a model, an agent, or a dependency changes. Pass rate shows whether results moved, and new frictions show where the agents’ behavior changed.

Prerequisites

Steps

1

Choose a fixed task set

Pick the tasks that you’ll run every time, and change them as little as possible, because a change to a task changes what you measure. Tag them, for example regression, so that you can find them later.
2

Run a first experiment

Launch the tasks with the treatments and agents that you track. You compare later experiments with this one.
3

Launch again after each change

After you change your product, a tool, or a dependency, open the finished experiment and select Launch again. It launches a new experiment with the same tasks, treatments, agents, and settings, and each task uses its latest version. If tasks changed, the dialog says how many. To change the model or anything else first, select Edit first.
4

Schedule it

To catch changes that you don’t make, such as a new model or a provider update, select Schedule on the experiment and choose Repeat, the time, and the Time zone. The dialog shows the First launch and the runs that it Uses for each launch and for the next 30 days. Then select Create schedule.
5

Compare the experiments

Open the schedule from Automate → Schedules. Its Experiments tab lists every experiment that it launched, Runs lists their runs, and Frictions ranks their frictions. Open two experiments and compare their Matrix pass rates. In Analyze → All experiments, each scheduled experiment shows From schedule and the schedule’s name.
6

Look for new frictions

In the schedule’s Frictions tab, or in Analyze → All frictions, sort the list by First seen. A friction that first appears after a change points to what the change broke.

Get notified

A scheduled experiment that finishes without problems sends no notification. One with errors or failed Evals notifies you as usual, and Obss also tells you when a schedule needs attention: These notifications arrive in the app and by email. Turn them on or off with Experiment updates under Profile → Notifications.

Interpret the results

A stable result keeps about the same pass rate on each task and shows no new frictions.

From the CLI and the plugin

Your AI assistant can do every step with the Obss Plugin or the CLI. For example, preview and create a schedule for 9:00 AM on weekdays:
To rerun with the first experiment’s task versions instead of the latest, pass --versions experiment to obss experiments rerun.

Other ways to run it

  • Launch from a release pipeline. Run obss experiments rerun in your pipeline with a personal API key, then read pass and fail counts with obss results analyze. See Run an experiment in CI.
  • Track several task sets. Use one experiment and one schedule for each set. A project can have up to 50 active schedules.
  • Compare agents as they change. Schedule a cross-agent experiment, such as one from the Latest model shootout template.

Next steps