Skip to main content
Run the same tasks with and without your tool, and compare the pass rates. The difference is the tool’s lift.

Prerequisites

  • A project with at least one agent turned on under Manage → Agents.
  • Enough runs in your balance. Each combination of task, treatment, agent, and trial uses one run (Runs and billing).
  • The skill, MCP server, CLI, or SDK that you want to measure.

Steps

1

Write tasks that use your product

In Manage → Tasks, write tasks like the ones your users give agents, each with a rubric and the files and machine it needs. Don’t name your tool in the task, so that you measure whether the agent finds it. Write more than 6 tasks: Comparisons needs at least 6 tasks where one side did better, and tied tasks don’t count.
2

Add your tool

In Manage → Tools, add the tool. Enter any secret it needs, such as an API key, in the dashboard.
3

Create a treatment with your tool

In Manage → Treatments (Toolsets), create a treatment that contains your tool. Every project already has Raw Agent, the usual baseline, which runs the agent as its vendor ships it.
4

Launch the experiment

Select New experiment and choose your tasks, both treatments, and one or more agents. Under Baseline, choose Raw Agent. Set Trials to 2 or more, and launch the experiment.
5

Compare pass rates on the Matrix

Open the experiment’s Matrix with the Evals metric. Compare each of your treatment’s columns with the same agent’s Raw Agent column, and hover over a cell for the trial count. For example, 1 of 4 trials passed (25%) on Raw Agent and 3 of 4 (75%) on your treatment is a lift of 50 percentage points on that task (Pass rate and lift).Switch the metric to Treatment use to see how often the agent called your tool. It’s a lower bound, because calls inside scripts don’t count, and it’s blank for an SDK.
6

Check that the lift holds up

Open the Comparisons tab. The sentence at the top says whether your treatment did better than Raw Agent, and the block under it gives each side’s pass rate and the result. A Clear difference is safe to act on. For a Likely difference, select Run again to be sure · N runs to launch a small follow-up experiment first. When Insights ran, the Brief in the block lists the frictions seen with the side that did worse. Each agent’s own result is under All N comparisons. Learn more in Comparisons.
7

Find what still blocks agents

On the Frictions tab, choose Product under Where to fix to see changes to make in your tool, API, errors, or docs. Fix the top ones as described in Find and fix frictions, then select Launch again.

Interpret the results

Your tool helps when Comparisons shows a Clear difference in your treatment’s favor and Treatment use is high.

From the CLI and the plugin

Your AI assistant can run this guide with the Obss Plugin or the CLI. Only tool secrets stay in the dashboard, and the commands return the link where you enter them. For example, ask your AI assistant:

Other ways to run it

  • Start from a template. On Home → Templates, Skill lift runs React tasks with no skill, then with two versions of a skill. Stripe interface evals compares Raw Agent with the Stripe MCP server, CLI, and SDK, and with the CLI plus a best-practices skill, all in Stripe test mode. Learn more in Templates.
  • Compare versions of a tool. Put each version in its own treatment, as the Skill lift template does.
  • Compare interfaces. Put each interface in its own treatment, as the Stripe interface evals template does, to see which gives the most lift for each agent.
  • Combine tools. Put a CLI and a skill in one treatment to measure their effect together.
  • Check every agent. Add more agents to see whether the tool helps all of them. See Compare agents and models.

Next steps