Skip to main content
Can’t find your question? Email support@obss.app, or see Troubleshooting for a specific problem.

About Obss

It measures how well AI agents, such as Claude Code, Codex, and Cursor, do real work with your product, and what helps them. You build evals and custom benchmarks, compare agents and models on the same tasks, and find where agents fail. See What is Obss.
A local try runs the agent once, so it can’t tell you how often the agent succeeds. An experiment repeats the same tasks across agents, models, and treatments on fixed machines and files, records every trajectory, and marks each rubric criterion Pass or Fail. See How Obss works.
Skills, MCP servers, CLIs, and SDKs, and the docs, APIs, and files they use. Put the tools in a treatment and compare it with Raw Agent, the usual baseline, which gives no tools. See Measure a tool’s impact.
Files and a machine attach to a task, tools attach to a treatment (toolset), and you choose agents at launch. See Where files, machines, and tools attach.

Plans and runs

Yes, with no credit card required. It resets your organization’s runs each month, and unused plan runs don’t roll over. See Pricing and Runs and billing.
Subscribe for more runs each month from Organization settings → Usage & billing, or top up at any time for runs that never expire. Larger top-ups cost less per run. For volume discounts, email support@obss.app.
One for each combination of task, treatment, agent, and trial. For example, 3 tasks × 2 treatments × 1 agent × 2 trials uses 12 runs, and the New experiment footer shows Uses 12 runs. See Launch an experiment.
A run that finishes and fails its rubric counts. Errored or cancelled runs don’t count against your organization’s run balance.
The cost of running the experiment on Obss, including the cloud machines. Your providers bill model use separately, through your own API keys or subscriptions.
Yes. Every plan includes unlimited members, projects, experiments, tasks, treatments, and tools, and full access to the dashboard, the CLI, and the plugin.
Yes, at no extra cost. Members share your organization’s run balance. See Invite teammates.

Agents and models

Built-in agents such as Claude Code, Codex, Cursor, Antigravity, and Qwen Code. Each works only with certain providers: Which providers each agent uses lists them, and each agent’s page under Manage → Agents shows which of your providers it can use.Custom agents are in private beta. Email support@obss.app to enable them for your organization, and then you can add any Agent Client Protocol (ACP) agent. See Custom agents.
Yes. Claude Code can use Anthropic (Claude Subscription), which you connect under Profile → Integrations. See Which providers each agent uses.
Yes, for Evals and Insights: connect Anthropic (Claude Subscription) under Profile → Integrations, then choose a Claude model under Obss AI features. Ask Obss and Agent mode can’t use Anthropic (Claude Subscription) models. See Obss AI features.
Organization API keys, gateways such as OpenRouter and Vercel AI Gateway, and OpenAI- or Anthropic-compatible endpoints. Each person can also connect personal subscriptions under Profile → Integrations. See Model providers.
Yes. Choose more than one model or effort level for an agent at launch, and each choice gets its own Matrix column. See Compare agents and models.
The limit on how many agent sessions run at once in a launch. It doesn’t control Evals or Insights. See Set Max Concurrency.

Evals, Insights, and results

Evals score each run against your rubric, and Insights read the trajectory to find frictions. They run independently.
Yes. Write a rubric of plain-language criteria, and Evals mark each one Pass or Fail. See Tasks and rubrics.
Not always. Evals and Insights read as much of the trajectory as fits the model’s context window (Evals also read the changed files), and the result says when something was left out. On older runs, a missing note doesn’t mean that the whole trajectory was read. See What Evals read and What Insights read.
A mode of Evals and Insights in which an Obss agent reads the whole run before it answers. For Evals, the agent checks the claims of the agent that did the run before it scores. For Insights, it checks each friction against what the tools returned, and on Comparisons it shows what a failed run and a passed run did differently. Agent mode costs more and takes a few minutes per run. It is in private beta. Email support@obss.app to enable it for your organization. See Agent mode.
Yes. Edit the rubric and select Rerun evals on the experiment to rescore the same recorded work. If the task itself changed, launch a new experiment. See Run Evals again.
On Matrix, with the Evals metric, compare each treatment’s cells with the baseline’s (chosen at launch) for the same agent and task. See Pass rate and lift, and check whether a difference holds up on Comparisons.
A short note under each difference on Comparisons: one sentence on what went wrong for the side that did worse, and the frictions seen in its runs. Obss counts every number, and the Experiment insights model writes only the sentence. It needs Insights, and it never names a cause. See Brief.
Yes. Open a run: Traces shows the Timeline and transcript, Output shows the final answer and changed files, Evals shows each criterion’s verdict, and the Run profile rail shows treatment use and where the time went. See Run details.
Yes. On the Runs tab, select two or three runs and select Compare. See Compare runs side by side.
Yes. A run’s ⋯ menu saves the trajectory, the output files, or the changes as a diff. The CLI and the plugin read runs, run profiles, trajectories, Evals, Insights, and frictions, and obss data query returns rows for your own analysis. See Run details.
Yes. Anyone with a public experiment link can view the results without an account. For a teammate, invite them to the organization instead. See Share results.

Experiments

Yes. Home has templates such as Getting started, Stripe interface evals, Latest model shootout, and Skill lift. See Templates.
Yes. On a finished experiment, select Launch again to run the same tasks, treatments, and agents, or Edit first to change something before you launch. See Launch again.
Yes. Select Schedule on a finished experiment, or open Automate → Schedules, and set the repeat and time zone. Each launch is a new experiment that uses runs. See Schedules.
Yes. Select Cancel experiment while it runs. Unfinished runs stop, completed runs stay available, and cancelled runs don’t count against your run balance. See Cancel an experiment.
Yes, from public github.com repositories, but not private ones. Use From GitHub to import, and Check for updates later. See Tools.
Yes. Tag tasks, files, machines, tools, treatments, and experiments, then filter any list with Tags. Tags don’t change a run. See Tags.

Ask Obss, the plugin, and the CLI

The in-app assistant for your project: it can create tasks, launch experiments, and explain results. Give it a model from OpenAI (ChatGPT Subscription), Nous Portal, or an organization provider under Organization settings → Obss AI features. It can’t use Anthropic (Claude Subscription). See Ask Obss.
Yes. With the Obss Plugin, your AI assistant (such as Codex, Claude Code, Cursor, or GitHub Copilot) can create tasks and treatments, launch and watch experiments, and read results. Launches use runs, so turn on tool approval in your AI assistant’s settings to have it ask you first. See Install the Obss Plugin.
You enter secrets, such as OAuth sign-ins, API keys, and secret headers, only in the dashboard. The CLI, the plugin, and Ask Obss give you a link to the exact page instead. To list every link, run obss setup links or ask for Get secure setup links (get_setup_links).
Yes: launch and watch experiments from the CLI in your own automation, signed in with a personal API key in OBSS_API_KEY. There is no built-in CI pipeline gate. See Run an experiment in CI.
Switch projects in the header, use Organization settings for shared setup, and use Profile for integrations, API keys, and notifications. The Dashboard map shows where everything is.