# Quickstart - Obss

Get started
Quickstart
Launch your first experiment from a Home template, then read its Matrix, Frictions, and runs.
Setup takes a few minutes, and the runs take several more after you launch. You can leave the page while they finish.
Prerequisites
An Obss account. Every organization starts on the
free plan
, which resets its runs each month.
Access to models through an API key (such as OpenAI or Anthropic), a gateway (such as OpenRouter), or a personal subscription (such as ChatGPT or Claude).
Model providers
lists them all.
Optional: the
CLI
or the
Obss Plugin
, to follow the
CLI
or
Plugin (MCP)
tabs.
Run your first experiment
1
Sign in
Go to
obss.app
, select
Get started
, and create an account or sign in. You land on
Home
in
My first project
, a
project
that Obss creates for you.
2
Connect a model provider
On
Home
, select
Connect models to get started
(or open
Organization settings → Model providers
) and connect an API key or a gateway. Connect a personal subscription, such as
OpenAI (ChatGPT Subscription)
or
Anthropic (Claude Subscription)
, under
Profile → Integrations
instead.
When the provider lists its models,
Home
shows
New experiment
.
You enter secrets, such as API keys and OAuth sign-ins, only in the dashboard. The CLI, the plugin, and Ask Obss can’t accept them.
3
Check agents and the default model
Open
Manage → Agents
. Built-in agents are on by default, so check that at least one shows models, such as Claude Code with an Anthropic key or Codex with an OpenAI key.
Agents
lists which providers each agent supports.
Then open
Organization settings → Obss AI features
and check that
Default model
has a model and no warning. Obss’s AI features (Evals, Insights, and Ask Obss) use this model from your providers unless you choose another under
Advanced
, and Obss can choose it for you when your first provider connects. If you plan to use
Ask Obss
, also check that the
Ask Obss model
meets its
prerequisites
.
4
Start from a template
Dashboard
CLI
Plugin (MCP)
On
Home
, under
Templates
, select one of these:
Template
What it does
Runs with one agent
Getting started
An agent inspects, fixes, and documents code. Three tasks, one treatment, one trial.
3
Skill lift
The same React tasks run with no skill, then with version 1 and version 2 of a skill. Three tasks, three treatments, three trials.
27
Getting started
is fastest, but its only treatment is
Raw Agent
, so it can’t show
lift
(the change in pass rate that a tool causes).
Skill lift
shows lift for a skill. Both have three tasks, so
Comparisons
shows
No clear difference
for them. Use them to learn the screens, and add tasks before you judge a skill.
Obss adds the template’s tasks, tools, and treatments to your project and opens
New experiment
with them selected. Learn more in
Templates
.
Templates are dashboard-only, so from the CLI you create a task instead. Install the CLI and sign in:
curl
-fsSL
https://obss.app/install
|
sh
obss
login
Copy the example
task.json
from
obss resources apply
, change its instructions and rubric, and apply it.
obss schema resources apply
lists every field.
obss
resources
apply
--operation-file
task.json
obss
catalog
list
--kind
task,treatment,agent
--projection
full
obss catalog list
returns the IDs your launch plan needs.
Templates are dashboard-only, so ask your AI assistant to create a task instead:
In my Obss project, create a task that asks the agent to add input
validation to a small function, with a rubric of three yes/no criteria.
It calls
Create or update a resource
(
author_resource
) and
Search the project catalog
(
inspect_catalog
).
The new tasks appear under
Manage → Tasks
.
5
Launch the experiment
Dashboard
CLI
Plugin (MCP)
In the
Configuration
panel, review
Tasks
,
Agents
,
Treatments (Toolsets)
, and
Trials
, and keep
Evals
and
Insights
on. The footer shows the runs the launch uses and how many you have left, for example
Uses 3 runs
. Select
Launch
.
Write a launch plan with the task’s version ID, the
Raw Agent
treatment, one agent, and one trial, starting from the
example plan
. Then check it, launch it, and wait:
obss
experiments
preflight
--plan-file
plan.json
obss
experiments
launch
--plan-file
plan.json
obss
experiments
watch
exp_8f2c41d0
launch
prints a launch ID and an experiment ID, and
watch
accepts either. For every plan field, run
obss schema experiments launch
or read
obss experiments launch
.
Ask your AI assistant to launch it:
Launch an experiment with that task, the Raw Agent treatment,
one agent, and one trial. Tell me when it finishes.
It calls
Check an experiment plan before launch
(
preflight_experiment
),
Launch an experiment
(
launch_experiment
), and then
Get experiment status
(
get_experiment_status
). A launch spends runs, so
turn on tool approval
in your AI assistant’s settings to have it ask you first.
Each run counts against your organization’s run balance. Learn more in
Launch an experiment
.
6
Read the Matrix
When the runs finish, open the experiment on
Matrix
, a heat map of pass rates for each task, agent, and treatment. On the
Evals
metric, hover over a cell to see the trial count, such as “2 of 3 trials passed”. The metric control also shows cost, tokens, duration, and more.
With
Skill lift
, compare each skill version’s cells with the
Raw Agent
cells for the same task. Matrix shows no lift number: the difference between the cells is the
lift
.
This
Matrix
is from the larger
Stripe interface evals
template, turned with
Transpose
so each row is an agent and treatment and each column is a task.
From the CLI, run
obss results analyze summarize_scope --experiment-id exp_8f2c41d0
. From the plugin, ask for a summary: it calls
Analyze experiment results
(
analyze_results
).
7
Check Frictions and open a run
Select the
Frictions
tab. Each friction is one problem that agents hit, and it shows where to fix it. Then select
Runs
and open a run. Its
Traces
,
Output
, and
Evals
tabs show what the agent did, what it produced, and the verdict on each rubric criterion.
With
Skill lift
, a notice can flag runs that hit
machine or task frictions
. A problem in the task or its machine caused those, so don’t count their results against the agent or the treatment.
From the CLI, run
obss frictions list --experiment-id exp_8f2c41d0
. From the plugin, ask for the frictions: it calls
List a project’s frictions
(
list_frictions
).
You launched an experiment and read its Matrix, Frictions, and runs. To ask about the results in plain language, open
Ask Obss
from the dashboard header.
Use the CLI or the plugin instead
From your AI assistant:
follow
Try a first prompt
, or
build an eval set for your own product
.
From the terminal:
follow
Run an experiment from the terminal
.
Next steps
Tasks and rubrics
Write a task about your own product.
Measure a tool's impact
Compare your skill, MCP server, CLI, or SDK with a baseline.
Read results
Go deeper on Matrix, lift, and each run.
Set up your organization
Finish the one-time setup checklist.
Introduction
How Obss works
Powered by
This documentation is built and hosted on Mintlify, a developer documentation platform