Skip to main content
An experiment runs every combination of its tasks, treatments, and agents once per trial, and each of those is one run. These commands launch experiments from a JSON plan, follow them, and share their results. To decide what goes in one, see Launch an experiment.

Launch plan

A launch plan is one JSON object that names every task version, treatment, and agent, so the same plan always launches the same work. preflight and launch both read it, and obss schema experiments launch lists every field and limit. To build a plan, run obss experiments clone-plan on an experiment you already ran, and edit it. For your first experiment, copy the example below and replace its IDs with the IDs from obss catalog list --projection full. For a baseline, use the trt_ ID of the project’s Raw Agent treatment.

Example

This plan runs 2 tasks × 2 treatments × 2 agents × 2 trials, which is 16 runs:

Fields

A plan can hold at most 100,000 runs. Keep any extra fields that clone-plan adds, such as a treatment’s contentHash.

obss experiments preflight

Check a launch plan without launching it or spending runs.
Plugin (MCP) tool: preflight_experiment Flags Pass exactly one of --plan or --plan-file. Also takes the global flags. Examples
Result fields Notes Preflight exits 0 whether or not the plan is ready, so a script either checks ready or passes --require-ready. A plan with evalsMode or insightsMode set to agent gets the blocker agent_mode_unavailable when your organization can’t use that Agent mode, or agent_mode_model_unsupported, with a setupUrl to Obss AI features, when a model can’t run it. A plan that doesn’t match the schema fails with a usage error (exit code 2) that names the first wrong fields, and details lists every issue.

obss experiments launch

Launch an experiment from a plan.
Plugin (MCP) tool: launch_experiment This command spends runs from your organization’s balance and doesn’t ask you to confirm, so run preflight first. In Ask Obss, you approve each call. Flags Pass exactly one of --plan or --plan-file. Also takes the global flags. Examples Launch, then watch until the experiment ends:
Result fields Notes obss experiments status and obss experiments watch take either ID. To retry after a lost response, send the same plan with the same launchAttemptId, and you get the first launch back. A refused launch exits with code 4, and error.code is one of these: In a pipeline, build a new launchAttemptId for each launch, such as from the pipeline run number, and reuse it only to retry that launch. A saved plan file keeps its old launchAttemptId and revisions, so launch the plan that preflight returns, not the file.

obss experiments status

Read an experiment’s progress once.
Plugin (MCP) tool: get_experiment_status. For an adm_ launch ID: get_launch_status. Arguments Flags Also takes the global flags. Examples
Result fields Notes When terminal is true, read the final runs with obss runs list. If you run obss runs list while a --wait-ms status read is still waiting, it can answer first with an older state.

obss experiments watch

Wait until an experiment ends, then print its final status.
Plugin (MCP) tool: get_experiment_status, get_launch_status. MCP hosts poll the status with wait_ms. Arguments Flags Also takes the global flags. Examples
Result fields When terminal is true, the command returns the same fields as obss experiments status: experiment, progress, terminal, and pollAfterMs. Notes Each read waits in the server for a change, and the command prints a progress line on stderr after each one unless you pass --quiet. Right after a launch, it also waits the few seconds an experiment can take to appear. On a timeout, the message is Experiment did not reach a terminal state after <n> seconds., and error.details has the id you watched, the experimentId when known, and the last status read. The experiment keeps running; watch it again. For pass counts, run obss results analyze after the watch ends.

obss experiments list

List and filter the experiments in the project.
Plugin (MCP) tool: list_experiments Flags The first eleven flags are the run scope flags except --experiment-id. Also takes the global flags. Examples List the experiments tagged for two releases:
List what a schedule launched:
List the experiments whose runs gave the agent an MCP server:
Result fields Unlike obss runs list, the result has no scope. Notes A run scope flag lists the experiments that have at least one run in that scope: the experiments of the runs that obss runs list returns with the same flags. --schedule-id also lists the experiments that the schedule launched before they have runs. An ID that the project doesn’t have fails with scope_not_found, and an unknown tag fails with tag_not_found, with details.tags naming it.

obss experiments get

Read an experiment’s progress, where it came from, who launched it, and the runs it used.
Plugin (MCP) tool: get_experiment Arguments Takes only the global flags. Examples
Result fields Notes A deleted experiment or schedule in lineage shows as deleted.

obss experiments brief

Read an experiment’s Brief: each clear or likely difference as Comparisons shows it, with one written sentence and the frictions seen with the side that did worse.
Plugin (MCP) tool: get_experiment_brief Arguments Flags Also takes the global flags. Examples Read the Brief:
It prints the sentence that opens Comparisons, then each difference: its block as obss evals compare prints it, Brief: and the sentence, and one line per friction with the failed runs of the side that did worse and the passed runs of the other side. A Likely difference prints its next step and command on its result line, and a Clear difference with a friction to fix in the product prints After a fix, run the N tasks that differed again · N runs with its command. After the differences come the state message, if any, what the differences leave out, the link to Comparisons, and Notes. With Insights in Agent mode, each difference ends with What the runs did differently. Each task has one line for the failed run and one for the passed run, with the step lines they quote and the run IDs. Once the insights agent has read every pair, the Brief: sentence says what the runs did across the tasks, and the command prints the example task with “Like N other tasks”, then each task that went differently with “Went differently”. Before that, it prints the first three tasks, and “Reading N more tasks…” while the agent is still reading. Last, it says how many tasks are left, with the --output json command that returns them all:
Print one difference as an issue in Markdown, the text that Copy as issue copies:
Read the differences and their counts as JSON:
Result fields Each finding has these fields: Each pair in explanations has these fields: worse and better each have these fields: Notes The command changes nothing. Obss chooses the differences and counts every number from the experiment’s Evals verdicts and frictions. A model writes only each sentence and, in Agent mode, each pair’s line. A friction listed with a difference was seen with the side that did worse. That does not mean it caused the difference. In Agent mode, a pair says where two runs went different ways, not why the difference happened. What the insights agent used to read the pairs counts in the Insights attempt’s agentUsage (obss insights list).
  • The findings are the differences that Comparisons shows as blocks. A No clear difference, a result for whether a treatment helps one agent more, and one agent’s row of a difference already shown for all agents are never findings.
  • A friction is listed with a difference only when at least 3 counted runs of worseSide have it, on at least 2 tasks, at a higher rate than the other side. See Brief.
  • In Agent mode, only runs that Evals marked Pass or Fail are paired, one pair per task.
  • Public experiment links don’t show the Brief.

obss experiments clone-plan

Get a launch plan from an experiment, ready to edit and launch.
Plugin (MCP) tool: get_experiment_plan Arguments Flags Also takes the global flags. Examples Save a plan, change it, check it, and launch it:
Plan the runs that check one likely difference again, check them, and launch them:
Result fields Notes The command changes nothing. Text output only counts the version changes, so use --output json to get the plan. --comparison takes the cmp_ ID of a Likely difference from obss evals compare, and the plan runs that likely difference again. Launching the plan uses runs from your run balance.
  • Tasks: The plan uses the comparison’s tasks, except tasks that already passed in every run, or failed in every run, at least 4 times on both sides together.
  • Sides: A treatment comparison keeps its agent, or every agent for all agents together. An agent comparison keeps its treatment. A comparison of whether a treatment helps one agent more (interaction) keeps both treatments and both agents.
  • Runs each: Each combination gets enough runs that the new experiment will likely settle the difference: from the experiment’s own (at least 3) up to 10.
  • Versions: The plan keeps the experiment’s task versions unless you pass --versions latest. Treatments use their current tools, as with Launch again.
  • Evals: Evals is on, with the mode, Evals model, and instructions of the experiment’s latest Evals attempt (the one that found the difference), even when the experiment launched with Evals off.
  • Name: “Run again:”, followed by the two sides and what they share, for example “Run again: Stripe CLI vs Raw Agent, Codex · gpt-5.5 · high”. For all agents together, the name ends with the two sides.

obss experiments rerun

Launch a finished experiment again in one call.
Plugin (MCP) tool: rerun_experiment This command spends runs from your organization’s balance. In Ask Obss, you approve each call. Evals and Insights run in the modes the experiment launched with, including Agent mode. If your organization can no longer use that Agent mode, the launch is refused with agent_mode_unavailable. To launch in Standard mode, plan the launch with clone-plan and set evalsMode or insightsMode to standard. Arguments Flags Also takes the global flags. Examples
Notes The new experiment has the same name, tasks, treatments, agents, trial count, Evals, and Insights, with each treatment’s current saved tools. If a task or treatment it used was deleted, the command refuses rather than launch less. To change the plan first, use clone-plan, edit it, and launch it. Result fields The same fields as obss experiments launch: launchId, experimentId, status, runCount, and dashboardUrl. The text output also prints the obss experiments watch command for the launch.

obss experiments cancel

Stop the unfinished work of an experiment, or of one run.
Plugin (MCP) tool: cancel_experiment
Cancelled runs stop and don’t finish. Completed work stays available.
Arguments Flags Also takes the global flags. Examples
Result fields

obss experiments publish

Publish a completed experiment at a public link that anyone can open without an account.
Plugin (MCP) tool: publish_experiment This command makes results public. In Ask Obss, you approve each call. Arguments Flags Also takes the global flags. Examples
Result fields Notes The page includes runs, traces, outputs, Evals, Insights, and text files up to 256 KiB. Larger files, binary files, and private machine settings aren’t shared. See Share results. Later results go live only when you publish an update.

obss experiments publications list

List your organization’s public links.
Plugin (MCP) tool: list_experiment_publications Flags Also takes the global flags. Examples

obss experiments publications get

Read one public link, its status, and the version it shows.
Plugin (MCP) tool: get_experiment_publication Arguments Takes only the global flags. Examples Read the lockVersion before you update or unpublish:
Result fields

obss experiments publications update

Publish the latest results to an existing public link.
Plugin (MCP) tool: update_experiment_publication This command makes results public. In Ask Obss, you approve each call.
The update replaces the results at the public link. The current page stays live until the new version is ready.
Arguments Flags Also takes the global flags. Examples

obss experiments publications unpublish

Stop sharing a public link.
Plugin (MCP) tool: unpublish_experiment
The public link stops working immediately for everyone who has it.
Arguments Flags Also takes the global flags. Examples
Result fields Notes A project with public links can’t be deleted until you unpublish them. Read results from a public link, one section at a time. You do not need to sign in.
Plugin (MCP) tool: get_public_experiment Arguments Flags A key that a section does not take fails with a usage error that names the keys it takes. Also takes the global flags. Examples
Notes Read the keys for other sections from overview, runs, and matrix, and page with --cursor while hasMore is true. A page that’s still being built, failed, or was unpublished returns its status instead of items. When the public page limits your reads, the error says how many seconds to wait. A large frictions result arrives as numbered parts (part.index of part.count) that share one selection. Read every page, then join the parts in order by appending each part’s clusters (a repeated clusterKey continues the previous cluster), family clusters, and outcome badge audits.

Progress counts

obss experiments status, watch, and get return progress, which counts each run in exactly one state. For each run’s exact outcome, use obss runs list.

Next steps