A
Agent
The AI agent that does the work in a run, such as Claude Code or Codex, with a model and an effort level. It can be built-in or a custom agent, and you choose it at launch. See Agents.Agent behavior
A Where to fix value: the evidence points to the agent’s own behavior, not to your product or setup. See Where to fix.Agent default
The model choice that lets a custom agent use its own default model. See Custom agents.Agent mode
The mode of Evals or Insights in which an Obss agent reads the whole run before it answers. Obss’s evals agent checks the claims of the agent that did the run before it scores. Obss’s insights agent checks each friction against what the tools returned, and on Comparisons it shows what a failed run and a passed run did differently. Agent mode is in private beta. Email support@obss.app to enable it for your organization. See Agent mode.AI assistant
The tool you work in, such as Codex, Claude Code, Cursor, or GitHub Copilot. In MCP terms, it’s the MCP host. It operates Obss through the Obss Plugin or the CLI. The same product can also be an agent, but the roles differ: your AI assistant launches experiments, and agents do the tasks.All experiments
Every experiment in the project, under Analyze. See Read results.All frictions
Every friction in the project, under Analyze, ranked by how many runs never got past it. See Frictions.All official treatments
All official skills, All official MCPs, All official CLIs, and All official SDKs: treatments Obss keeps for you, each with the official version of every project tool of that kind. See Treatments (Toolsets).All runs
Every run in the project, under Analyze. See Read results.API key
A personal key from Profile → API keys for places where you can’t sign in through a browser, such as CI. The CLI reads it fromOBSS_API_KEY, and it acts as you until you revoke it. Organization API keys can’t use the CLI, and the plugin signs in with OAuth. See Profile and API keys.
Ask Obss
The in-app assistant for your project, with the same capabilities as the plugin and the CLI. You give it a model under Organization settings → Obss AI features. See Ask Obss.Ask Obss in Slack
Ask Obss inside Slack, where the Obss app answers direct messages and channel mentions. See Use Ask Obss in Slack.Assets
The earlier name for Tools.B
Baseline
The treatment you compare the others with, chosen under Baseline at launch. It’s usually Raw Agent, which gives no tools. See Baseline and lift.Brief
The note under a difference on Comparisons: one sentence on what went wrong for the side that did worse, written by the Experiment insights model, and the frictions seen in its runs. It never names a cause. Each Brief has abrf_ ID. See Brief.
C
Check for updates
Compares a file or skill imported from GitHub with its current source, so you can update it. See Tools.CLI
- The Obss CLI (
obss), installed from Automate → Obss CLI. See Use the CLI. - A CLI tool: a command-line tool you give an agent in a treatment, under Manage → Tools → CLIs. See Tools.
Compare
A side-by-side view of two or three runs across Traces, Output, and Evals. See Compare runs side by side.Comparisons
The experiment tab that says who did better, how sure that is, and what to do next. It opens with one sentence that answers the experiment’s question and a block for each difference, then All N comparisons. Each result is a Clear difference, a Likely difference, or a No clear difference (which doesn’t mean the two are the same). You can run a Likely difference again to be sure. Each comparison has acmp_ ID. See Comparisons.
Criterion
One item in a rubric that a run passes or fails, added with Add criterion. See Tasks and rubrics.Custom agent
An Agent Client Protocol (ACP) agent your organization adds, which Obss installs in each run’s machine and runs like a built-in agent. It shows a Custom tag and a status: Needs values, Needs attention, Testing, Not tested, or Ready. Custom agents are in private beta. Email support@obss.app to enable them for your organization. See Custom agents.D
Default
The effort choice that sends no effort level, so the agent uses its own default for the model. Cursor is the exception: it uses the default variant Cursor lists for the model. See Agents.Download
The items in a run’s ⋯ menu that save recorded work: Download trajectory, Download output files, and Download changes (diff). See Run details.E
Edit first
Opens New experiment with a finished experiment’s tasks, treatments, and agents, so you can change them before you launch. See Launch an experiment.Effort level
How much reasoning an agent’s model uses, where the model supports it, chosen per agent at launch. Levels can include none, minimal, low, medium, high, xhigh, and max. See Agents.Eval set
The tasks, machine, files, tools, and treatments that evaluate one product. The Obss Plugin skills build one from a product’s docs. See Build an eval set and read its results.Evals
How Obss scores a finished run. The Evals model reads the trajectory and changed files and marks each rubric criterion Pass or Fail, and the Evals tab notes when it couldn’t read the whole run. Evals run independently of Insights. See Evals.Evals mode
How Evals review each run against the rubric. Standard scores each run in one fast, low-cost pass with your Evals model. Agent (private beta) uses Obss’s evals agent for an in-depth review that costs more and takes a few minutes per run. See Agent mode.Evals model
The model that scores runs for Evals, chosen under Organization settings → Obss AI features. See Obss AI features.Experiment
One launch of tasks, treatments, agents, and trials, where each combination is one run. It saves what it launched, so later edits to a task or treatment don’t change its results. See Experiments.Experiment insights model
The model that groups every run’s Insights findings into frictions, and writes the Brief. See Obss AI features.F
Family
The cause of a friction: Product, Docs & skills, Machine, Agent software, Agent reasoning, or Task quality. Each friction has one. See Family.Files
Folders you attach to a task, which Obss puts in the agent’s working directory before the run starts. Create them under Manage → Files, by upload or from GitHub. See Machines and files.Friction
One problem that agents hit, tracked across runs and experiments with a stablefrc_ ID, one family, and one place to fix it. Most come from Insights. A run that fails before the agent answers also counts toward a friction for its cause. See Frictions.
From GitHub
Imports a file or skill from a publicgithub.com repository, which you can later compare with Check for updates. See Tools.
I
ID
A record’s stable public identifier, with a prefix for each kind:exp_ experiments, tsk_ tasks, trt_ treatments, frc_ frictions, sch_ schedules, and tag_ tags. The CLI and the plugin use these IDs. See IDs and limits.
Inactive
A file or skill no longer offered for new tasks or treatments, but kept visible so past results make sense. A GitHub import becomes inactive when Check for updates finds its source removed. See Tools.Insights
The analysis of a finished run’s trajectory, in which the Insights model finds frictions and patterns. Insights run independently of Evals. See Insights.Insights mode
How Insights review each run. Standard reviews each run in one fast, low-cost pass with your Insights model. Agent (private beta) uses Obss’s insights agent for an in-depth review that costs more and takes a few minutes per run. See Agent mode.Insights model
The model that finds the frictions and patterns in each run. See Obss AI features.Integrations
Your personal connections under Profile → Integrations: personal provider subscriptions, GitHub, Ask Obss, and MCP server sign-ins. Select Disconnect to stop using a connected account. See Profile and API keys.L
Launch again
Launches a new experiment with a finished experiment’s tasks, treatments, agents, and settings, using each task’s latest version. CLI:obss experiments rerun. Plugin: Launch an experiment again (rerun_experiment). See Launch again.
Launch now
Launches a schedule’s experiment right away, under your account, without moving its next launch. See Schedules.Launch plan
An experiment’s exact selection as the CLI and the plugin send it: task versions, treatments, agents, trials, and settings. Copy a finished experiment’s plan withobss experiments clone-plan or get_experiment_plan to change and relaunch it. See obss experiments launch.
Lift
The difference in pass rate between a treatment and the baseline on the same tasks and agents. See Baseline and lift.M
Machine
The environment a run executes in, set on the task. The default is the Standard Dev Machine, and you create others under Manage → Machines (Environments). See Machines and files.Machine and task frictions
Frictions in the Machine or Task quality family. They come from the experiment’s own machine or task, not the product you measure, so check them before you trust the results. Don’t confuse them with Setup under Where to fix. See Machine and task frictions.Main comparisons
The comparisons that answer an experiment’s question, such as each treatment against the baseline for all agents together. Comparisons shows them first, and checks the others (the breakdowns) more strictly. See Main comparisons and breakdowns.Matrix
An experiment’s first tab: a heat map of tasks by agent and treatment. Each cell shows the chosen metric: Evals (the pass rate), Steps, Tool calls, Treatment use, Frictions, Tokens, Cost, or Duration. See Matrix.Max Concurrency
How many agent sessions can run at once in a launch. It doesn’t control Evals or Insights. See Launch an experiment.MCP
- The Obss MCP server: how your AI assistant operates Obss, through the Obss Plugin. See the Plugin (MCP) reference.
- An MCP server tool: an MCP server you give an agent in a treatment, under Manage → Tools → MCP servers. See Tools.
Members
The people in your organization, who share its projects, model providers, agents, and run balance. Manage them under Organization settings → Members. See Set up your organization.Metrics
A run’s steps, tool calls, tokens, cost, and duration: how hard the agent worked, not only whether it passed. See Read results.Model providers
Your organization’s API keys and gateways under Organization settings → Model providers, and the personal subscriptions that each member connects under Profile → Integrations. Agents and Obss AI features use their models. See Model providers.N
Notifications
Your in-app notifications, under the header bell. Set Email and In-app delivery per type under Profile → Notifications. See Notifications.O
Official version
The one version of a tool that the project marks as part of its intended interface, shown with an official mark and used by the All official treatments. See Tools.Obss AI features
Your organization’s model choices for Obss’s own work: the Evals model, Insights model, Experiment insights model, and Ask Obss model. You can add custom instructions for Evals, Insights, and Experiment insights. See Obss AI features.Obss Plugin
The package for your AI assistant that connects the Obss MCP server and adds skills for building eval sets, launching experiments, and reading results. See Install the Obss Plugin.Organization
Your company’s account, which holds its projects, members, model providers, and run balance. Change it under Organization settings. See Set up your organization.Output
The run tab for what the agent delivered: its final message under Answer, and changed files with diffs under Files. See Run details.P
Parent experiment
The experiment that another one was launched from, with Launch again, Edit first, or a schedule. Experiment lists show it as Parent: and its name, and Open parent experiment opens it. The experiments launched from it are its child experiments. See Launch an experiment.Pass rate
The share of runs that passed the task’s rubric. Hover over a Matrix cell to see the count, such as “3 of 4 trials passed”. See Pass rate and lift.Pause
Stops a schedule’s launches until you select Resume. Obss also pauses a schedule that needs a fix and marks it Needs attention. See Schedules.Plan runs
The runs your free or paid plan gives your organization each billing period. Unused plan runs don’t roll over. See Runs and billing.Project
The place that holds your tasks, files, machines, tools, treatments, experiments, and schedules. An organization can have many, and you switch between them in the header. See Projects.Public experiment link
A link that lets anyone view an experiment’s results without an account, created with Share on a finished experiment. Updates go live only when you publish them. See Share results.Public sharing
The Organization settings tab that lists every public experiment link in the organization. See Share results.Publish
Creates a public experiment link, or updates it with the latest results. CLI:obss experiments publish. Plugin: Publish an experiment (publish_experiment). See Share results.
R
Raw Agent
The treatment with no tools: the agent as its vendor ships it. It’s the usual baseline. See Treatments (Toolsets).Recovery
Whether the agent got past a friction: Recovered, Unrecovered, or Unknown. See Recovery.Repeat
When a schedule launches: every hour, day, weekday, week, or month, or on a custom cron expression. Schedules start on the quarter hour. See Schedules.Rerun evals
Rescores the same recorded work without rerunning the agent, such as after a rubric or Evals model change. Before Evals have run, the action is Run evals. See Run Evals again.Rerun Insights
Analyzes the same recorded work again without rerunning the agent. Before Insights have run, the action is Run insights. See Run Insights again.Rubric
The criteria that Evals check each run against to decide whether it passes the task. See Tasks and rubrics.Run
One agent’s attempt at one task with one treatment, and the unit you pay for. Errored or cancelled runs don’t count against your organization’s run balance. See Runs and billing.Run profile
The side rail on a run that shows treatment use, tool calls, what stood out, and where the time went. CLI:obss runs profile. Plugin: Get a run profile (get_run_profile). See Run profile.
Runs
- The Runs tab: the tab on an experiment, task, tool, treatment, machine, file, agent, or schedule that lists its runs.
- Run balance: the runs your organization can spend, from plan runs and top-up runs. See Runs and billing.
S
Schedule
A rule under Automate → Schedules that launches a finished experiment again on a repeat, as a new experiment that uses runs. It runs as whoever created it, resumed it, or last changed its repeat or experiment. See Schedules.SDK
A software development kit you give an agent in a treatment, created under Manage → Tools → SDKs. See Tools.Share
Opens the Share experiment dialog on a finished experiment, where you create a public experiment link or invite a teammate to view it. See Share results.Skill
A folder of instructions and files, with aSKILL.md, that an agent can read when a treatment includes it. Create skills under Manage → Tools → Skills. See Tools.
Standard Dev Machine
The default machine for a task: Ubuntu 24.04 with Node.js 24, Python 3.12, and Git. See Standard Dev Machine.Standard mode
The default mode of Evals and Insights, which reviews each run in one pass with your Evals model or Insights model. See Evals mode and Insights mode.T
Tag
A label with a stabletag_ ID that groups tasks, files, machines, tools, treatments, or experiments, so you can filter lists with Tags. Tags don’t change a run. See Tags.
Task
The work an agent attempts: instructions, a rubric, files, and a machine. Create tasks under Manage → Tasks. See Tasks and rubrics.Template
A ready-made experiment on Home. Selecting one adds its tasks, tools, and treatments to your project and opens New experiment. See Templates.Test agent
Launches one short run with the Raw Agent treatment (no tools) to check that a custom agent installs, starts, and answers. It uses one run from your balance. See Custom agents.Timeline
The top of the Traces tab, showing when tool calls happened, grouped by model turn when the run recorded turns. See Timeline.Tools
Skills, MCP servers, CLIs, and SDKs, under Manage → Tools. You attach them to treatments, not tasks. See Tools.Toolset
The set of tools in a treatment. See Treatment (toolset).Top-up runs
Runs you buy on top of your plan, which never expire. See Runs and billing.Traces
The run tab with the recorded trajectory: the Timeline, then the transcript grouped by model turn. See Run details.Transcript
A run’s recorded trajectory, shown on Traces grouped by model turn. To save it, choose ⋯ → Download trajectory on the run. See Run details.Treatment (toolset)
One set of tools you compare: the skills, MCP servers, CLIs, and SDKs an agent gets in a run. Raw Agent gives none. Create treatments under Manage → Treatments (Toolsets). See Treatments (Toolsets).Treatment use
Whether the agent used the treatment’s tools: Used, Not used, Not measured, Can’t tell, or Not applicable. It shows in the Run profile and the Runs tab. See Treatment use.Trial
One repeat of the same task, treatment, and agent, set at launch. Each trial is one run against your run balance. See Launch an experiment.U
Unpublish
Stops sharing an experiment, so its public link no longer works. See Share results.Upload
Adds a local folder or ZIP archive as a file or skill, or imports tasks from a ZIP archive. CLI:obss uploads commands. Plugin: Start an archive upload (create_upload). See obss uploads create.
