# Use the CLI - Obss

Operate
Use the CLI
Install the obss CLI, sign in, choose a project, and run experiments from your terminal, scripts, and CI.
The
obss
CLI can do everything the
Obss Plugin
and
Ask Obss
can, with stable JSON output for scripts, CI jobs, and AI assistants. You still read results and enter secrets in the dashboard. For every command and flag, see the
CLI reference
.
Install the CLI
curl
-fsSL
https://obss.app/install
|
sh
The dashboard shows the same install and sign-in commands under
Automate → Obss CLI
:
Sign in
obss
login
obss login
opens your browser and prints a sign-in link and a confirmation code. Confirm the code, and the CLI saves your sign-in. To sign in on another device, pass
--no-browser
and open the link there.
obss whoami
shows who you’re signed in as, and
obss logout
removes the saved sign-in.
Use an API key
Use a personal API key where nobody can sign in through a browser, such as in a CI job or on the machine that an agent runs in.
1
Create a personal API key
In the dashboard, open
Profile → API keys
and
create a key
.
2
Set the key where the CLI runs
Store the key as a CI secret exposed as
OBSS_API_KEY
.
export
OBSS_API_KEY
=
"<your personal API key>"
obss
whoami
obss whoami
confirms that the CLI is using an API key.
While
OBSS_API_KEY
is set:
Every command uses the key, which acts as you in its organization, instead of your saved sign-in.
obss login
doesn’t sign in. Unset the variable to sign in with your browser.
obss logout
removes only the saved sign-in, and commands keep using the key.
The CLI doesn’t update itself. Run
obss upgrade
when you want a newer release.
An invalid or revoked key gives an auth error (exit code
3
) that points to the
API keys
page. Organization API keys can’t use the CLI.
Choose a project
Most commands work on one project. List your projects and save the one to use:
obss
projects
list
obss
projects
use
prj_2b7e90c4
obss
projects
current
To create a project, give it a name and an attempt ID. Running the command again with the same attempt ID returns the same project.
obss
projects
create
--name
"Checkout agents"
--mutation-attempt-id
checkout-agents-1
To use a different project for one command, pass
--project-id
. It doesn’t change the saved project. Learn more in
obss projects
.
Run an experiment from the terminal
This workflow uses only CLI commands, so an AI assistant can run it too.
1
Find your tasks, treatments, and agents
obss
catalog
list
--kind
task,treatment,agent
--output
json
catalog list
returns each item’s stable ID and its current version ID. A launch plan uses each task’s version ID (
tskver_…
), not its task ID.
2
Write a launch plan
If you’ve already run an experiment, copy its plan and edit it:
obss
experiments
clone-plan
exp_8f2c41d0
--output
json
|
jq
'.data.plan'
>
plan.json
Otherwise, start from the
example plan
and fill in the IDs from the catalog.
The plan lists the task versions, treatments, agents, trials, and Evals and Insights settings. Give each new launch its own
launchAttemptId
, and reuse an ID only to retry the same launch.
obss schema experiments launch
shows every field.
3
Check the plan
obss
experiments
preflight
--plan-file
plan.json
Preflight
changes nothing and uses no runs. If a tool or machine still needs a value, such as an API key, it lists the dashboard link where you enter it.
4
Launch it
obss
experiments
launch
--plan-file
plan.json
The
launch
uses runs from your run balance. The CLI prints the launch ID, the experiment ID once it exists, the dashboard link, and the command to watch it.
5
Watch it finish
obss
experiments
watch
adm_5f1c9e20
watch
takes the launch ID (
adm_…
) or the experiment ID (
exp_…
), waits until the experiment ends, and prints its progress counts. To check once without waiting, use
obss experiments status
.
6
Read the results
obss
results
analyze
summarize_scope
--experiment-id
exp_8f2c41d0
obss
results
analyze
compare_treatments
--experiment-id
exp_8f2c41d0
obss
runs
list
--experiment-id
exp_8f2c41d0
obss
runs
profile
run_4c7e19a2
results analyze
summarizes pass and fail counts and compares treatments, and
runs list
pages through every run.
runs profile
shows where one run’s tool calls, time, and tokens went, and which treatment tools it used.
7
Read the frictions
obss
frictions
list
--experiment-id
exp_8f2c41d0
--fix-in
product
--recovery
unrecovered
obss
frictions
get
frc_7d20e6b4
obss
runs
list
--friction-id
frc_7d20e6b4
frictions list
ranks frictions by the number of runs in which the agent didn’t recover.
frictions get
reads one friction: what goes wrong, the suggested fix, where it happens most, and recent runs.
To run a finished experiment again, use
obss experiments rerun
. To run it on a schedule, use
obss schedules create
and
track the results over time
.
Run an experiment in CI
A CI job can launch an experiment, wait for it, and fail when the result falls below a threshold you choose. The CLI doesn’t decide pass or fail. Your job reads the counts and applies the threshold.
1
Set the API key and project
Store a
personal API key
as a CI secret named
OBSS_API_KEY
. A key has no saved project, so pass
--project-id
on every command.
P
=
prj_2b7e90c4
2
Stop if the plan is blocked
Keep a plan file in your repository.
obss
experiments
preflight
--plan-file
plan.json
--project-id
"
$P
"
--require-ready
--output
json
>
preflight.json
Preflight exits with code
0
even when the plan is blocked. With
--require-ready
, it exits with code
6
instead, which fails the job.
preflight.json
still lists the
blockers
and
issues
to fix.
3
Launch with a new attempt ID
A plan in your repository goes out of date when agents, models, or providers change, and the launch then fails with
configuration_changed
. Preflight returns a current copy in
data.plan
, so launch that. Give each pipeline run its own
launchAttemptId
, or you get the first launch back instead of a new experiment.
jq
--arg
id
"ci-
$CI_PIPELINE_ID
"
'.data.plan | .launchAttemptId = $id'
preflight.json
>
launch.json
LAUNCH_ID
=
$(
obss
experiments
launch
--plan-file
launch.json
--project-id
"
$P
"
--output
json
|
jq
-r
'.data.launchId'
)
If the launch is refused, read
error.code
. For example,
quota_exceeded
means your run balance doesn’t cover the plan, and a person must add runs.
4
Wait for the experiment to end
obss
experiments
watch
"
$LAUNCH_ID
"
--project-id
"
$P
"
--timeout-seconds
7200
\
--fail-on
completed_with_errors,cancelled,errored,admission_failed
--output
json
>
status.json
watch
exits with code
0
for every end state, including errored and cancelled. With
--fail-on
, it exits with code
6
when the experiment ends in a state you list, which fails the job.
status.json
still has the result.
completed
means every run finished, so the example lists every other end state. If the timeout passes first, it exits with code
4
and
error.code
watch_timeout
, and the experiment keeps running.
5
Fail the job below your threshold
EXP_ID
=
$(
jq
-r
'.data.experiment.experimentId'
status.json
)
obss
results
analyze
summarize_scope
--experiment-id
"
$EXP_ID
"
--project-id
"
$P
"
--output
json
>
summary.json
jq
-e
'.data.result.totals | (.passedRuns + .failedRuns) > 0 and (.passedRuns / (.passedRuns + .failedRuns)) >= 0.8'
summary.json
This fails the job when no run has a verdict, or when fewer than 80% of the runs with a verdict passed. The result is in
data.result
:
totals
has
runs
,
passedRuns
,
failedRuns
,
evalsFailedRuns
,
infraBlockedRuns
,
inconclusiveRuns
, and
pendingRuns
.
treatments[]
has
treatmentKey
,
runCount
,
passedRuns
, and
failedRuns
for each treatment.
Compute a pass rate as
passedRuns / (passedRuns + failedRuns)
. Blocked and inconclusive runs, and runs whose Evals failed, have no verdict.
To compare a treatment with the baseline in a script, run
obss results analyze compare_treatments
. Each treatment has
passedRuns
,
failedRuns
,
failureRate
, and
deltaFromReference.failureRate
, its difference in failure rate from the baseline.
If the tasks and agents don’t change,
obss experiments rerun exp_8f2c41d0 --attempt-id "ci-$CI_PIPELINE_ID"
is a shorter way to launch the same experiment again (see
attempt IDs
).
Script the CLI
JSON output:
Pass
--output json
to any command except
obss acp
, which writes only protocol messages. Stdout then has one envelope:
{"ok": true, "command": …, "data": …}
on success, or
{"ok": false, "command": …, "error": {"code": …, "message": …}}
on failure.
Exit codes:
Check the exit code, not the text:
0
success,
1
unexpected error,
2
usage error,
3
auth error,
4
API error,
5
network error, and
6
when the result matched
--fail-on
or
--require-ready
. See
Exit codes
.
Command contracts:
obss schema
lists every command, and
obss schema <command>
prints its flags and output as JSON Schema.
Confirmation:
A command that removes, cancels, disconnects, unpublishes, or replaces data asks you to confirm in a terminal. With
--output json
, or when there’s no terminal, pass
--yes
. Without it, the command exits with code
2
and changes nothing.
Retries:
Commands that launch or create take an attempt ID. If you run one again with the same attempt ID, you get the same result instead of a second launch.
Secrets:
The CLI never takes a secret on the command line. It returns the dashboard link instead, and
obss setup links
lists each machine and tool that still needs a value.
The
CLI reference
covers global flags, paging, and environment variables.
Keep the CLI current
After a command, the CLI checks for a new release (at most once an hour) and installs it in the background, so your next command uses it. To update now, run:
obss
upgrade
obss upgrade
doesn’t need a sign-in. The CLI doesn’t update itself in CI, while
OBSS_API_KEY
is set, or when run from source. The reference explains how to
turn off background updates
.
When Obss refuses an old release
When Obss no longer accepts the installed release, commands that contact Obss fail with exit code
4
and this message:
obss: This obss release is older than the server supports: run `obss upgrade`, then retry.
With
--output json
,
error.code
is
client_update_required
. The CLI reference has more on
refused releases
.
If you
run Ask Obss as a custom agent
from a pinned CLI download, it doesn’t pick up a newer release. Copy the
url
and
sha256
of a current release into the agent’s
Download URL
and
SHA-256
fields, save the agent, and test it again.
Next steps
CLI reference
Look up every command, flag, and environment variable.
Measure a tool's impact
Run a lift experiment end to end.
Install the Obss Plugin
Use the same capabilities inside your AI assistant.
Launch an experiment
Learn what goes into a launch plan.
Find and fix frictions
Turn the frictions you read into fixes.
Obss Plugin (MCP)
Ask Obss
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform