By obss team. Updated . Experiment figures identify their source snapshots and review dates below.
Agent usability testing tools for APIs, SDKs, CLIs, and MCP
Agent usability testing tools measure whether coding agents can complete real customer tasks through your API, SDK, CLI, or MCP server. Choose Obss for managed product experiments that check whether the agent reached the right outcome and show how it got there. In our ClickHouse export experiment, all 12 completed trials produced the correct file through their assigned interface. The traces still showed redundant page reads and error recovery.
Shortlist by the job. Compare Obss and Lightsage for managed coding-agent tests. Evaluate the open-source Gauge CLI if your team wants to run tests locally. Use MCPJam for MCP client behavior and authorization debugging, or MCP Inspector for a direct server check. These tools test different layers of the same workflow.
Run a known pass and a known fail
The Obss agent usability starter has a pinned Qdrant SDK, synthetic fixtures, and an independent outcome checker. It tests tenant-filtered search, complete paginated export, and persistence after reopening. Run both the reference solution and an incomplete starter before grading any agent output:
git clone https://github.com/Obss-Inc/agent-usability-starter.git
cd agent-usability-starter
git checkout --detach 498859279827cf6c9e111ff5fa215397988a4eed
docker build -t obss-agent-usability-starter:v1 .
python3 runner.py demo --output results/demo
cat results/demo/report.md
Expect six measured controls: three reference passes and three incomplete-starter failures. Build with Docker and run the command with Python 3.9 or later. Verification runs without network access or product credentials. It does not run a model. To inspect actual agent outputs and the grading disagreement, read the 32-attempt SDK documentation experiment.
The starter is an SDK fixture, not a four-interface or vendor benchmark. For the other surfaces, use the same customer task and decide what independent evidence would prove it succeeded.
Agent usability and agent observability in Agent Experience
Agent Experience, or AX, is how an agent discovers, understands, and uses a product on someone's behalf. For a developer product, that can mean finding the right documentation, installing an SDK, authenticating, choosing a tool, recovering from an error, and checking the result.
Agent usability asks whether an agent can complete a useful task with your product. Agent observability makes its behavior inspectable: the steps, tool calls, errors, retries, files, resource use, and final output that explain the result. In obss, those views come together in recorded experiment runs.
| Question | Evidence to collect |
|---|---|
| Can the agent discover the product or interface? | Product selection, documentation retrieval, and available tools |
| Can the interface execute correctly? | Requests, responses, permissions, errors, and resulting state |
| Can the agent complete the customer's task? | A verified outcome under realistic starting conditions |
| What happened during the attempt? | Ordered traces, retries, recovery, interface use, output, and resource use |
| Did a product change improve the experience? | Comparable baseline and revised conditions across repeated attempts |
A valid tool response establishes one part of that chain. An agent can still choose the wrong operation, miss a page of results, or finish through another product. Your evaluation needs both outcome checks and evidence of any required interface use.
This distinction between a transcript and the resulting environment state is also central to Anthropic's guidance on agent evaluations.
Evidence from Obss experiments
One ClickHouse export through API, SDK, CLI, and MCP
The same task passed 3/3 times through each interface, but the paths differed. In the November 16, 2026 pilot, Codex 0.167.0 with GPT-6.1-Sol at medium effort used the HTTP API, ClickHouse Connect Python SDK, native ClickHouse CLI, or official MCP server. Each attempt started with the same 240 synthetic orders in a fresh instance. The task: export exactly the 123 paid January orders for one tenant, with at most 40 order rows returned per query.
| Interface | Passed / completed | Setup blocked | Median agent time | Median input tokens |
|---|---|---|---|---|
| HTTP API | 3/3 | 0 | 33.1 s | 70,538 |
| Python SDK | 3/3 | 0 | 32.4 s | 69,396 |
| Native CLI | 3/3 | 3 | 32.6 s | 70,056 |
| Official MCP | 3/3 | 0 | 37.4 s | 109,883 |
Pass required an exact match to the fixture's order IDs and amounts, plus a trace check that the agent used the assigned interface and respected the 40-row limit. Three initial CLI attempts stopped during installation before Codex started. The three CLI attempts in the pass denominator used the native binary from the pinned ClickHouse image. Agent time excludes setup and independent grading. Input tokens are whole-run model totals.
- MCP: One agent copied each page into Python; the other two moved the rows into the CSV programmatically using the Codex tool runner. Two made a fifth page read that returned nothing after four pages of 40, 40, 40, and 3 rows. One HTTP and one SDK run also queried an empty page. Through HTTP, SDK, and CLI, the agent wrote a local program to pass query results directly into the file.
- HTTP: One successful run made two failed queries. It first saw a generic HTTP 500 in its Python exception, then printed the response body, found ClickHouse's
ILLEGAL_AGGREGATIONerror, fixed an alias, and completed the export.
For a high-volume export workflow, test whether a tool can return a file or a download handle and tell the caller when there is no next page. Compare verified outcome, page calls, tokens, and time on the same fixture before changing the interface. This pilot had one task, one model, and three completed trials per interface; it does not establish an interface ranking or isolate the source of the token difference.
Reproduce the fixture and inspect all 12 exported CSVs, the attempt ledger, and selected traces. Four reference clients independently cleared the checker before the agent runs. The checker also rejected a first-page-only export, a duplicate, and an incorrect amount.
Comparing product interfaces across 260 attempts
Our public RunType experiment crossed 13 tasks, five setups, two agent configurations, and two trials per combination: 260 attempts. The suite included live workflows, code integrations, and product-knowledge tasks. Here are the recorded passes across all 13 tasks:
| Product setup | Claude Code passes | Codex passes |
|---|---|---|
| Raw agent baseline | 20 / 26 | 21 / 26 |
| SDK | 23 / 26 | 18 / 26 |
| CLI | 19 / 26 | 18 / 26 |
| MCP | 20 / 26 | 19 / 26 |
| Skills | 18 / 26 | 18 / 26 |
Claude Code used Opus 5 with harness version 2.1.283; Codex used gpt-5.6-sol with harness version 0.158.0. The public snapshot reviewed here is version 3. These are descriptive outcomes for those tasks and configurations, with only two repeats per combination.
There is no single interface that improved both configurations across the suite. On the ticket-triage task, the rubric checked that the agent created a saved RunType flow or agent, ran all five tickets through it, and reported the returned results correctly. That makes successful use of the product part of the evidence.
A passing Vercel task still contained a failed file lookup
In the public Vercel MCP experiment, one historical passing attempt used a placeholder file ID, received a 404, then listed the files and passed the returned entry's uid as fileId. The inspected lookup sequence contained one extra failed MCP read beyond a valid list-then-read path.
The two model turns after the failed-request turn generated 995 output tokens and spanned 23.319 seconds, including a legitimate source-hash check. These are observed recovery-sequence totals, not proven avoidable overhead or an average across attempts. The annotated file-ID handoff and reusable checklist show how to turn a passing trajectory into a specific interface experiment. This is the practical role of agent observability in a usability test.
Observing how agents discover an API
In our analysis of 112 baseline vector-database trials, covering 57 Claude Code and 55 Codex trials across eight independent runs, neither agent performed a web search or fetched an external documentation page. They used model knowledge, workspace files, and runtime inspection of the installed package. Seven trials used SQLite instead of the intended vector database.
Those observations are specific to that benchmark. For your own SDK, inspect whether agents actually find your docs, read source, probe the package, or switch tools. Then improve the surface they encounter and test it again.
Agent Experience evaluation tools at a glance
| Tool | Useful starting point | What to establish in a pilot |
|---|---|---|
| obss | Agent usability testing and agent observability across product tasks, agents, and interface setups | Task coverage, recorded traces, friction diagnosis, and evidence behind verdicts |
| The Prompting Company | Hosted coding-agent tasks in isolated sandboxes | Task goals, agent/model environments, transcripts, artifacts, and failure analysis |
| ax-eval (chenmingtang830) | Reviewed API, CLI, SDK, and MCP task packs with live read-back verification | Product-specific packs, credentials, sandbox scope, harness support, and reset logic your team supplies |
| Microsoft Scope | Open-source Agent Experience experiments with criteria and run evidence | Available agent integrations and deployment requirements; Codex is listed as planned in the docs reviewed October 26 |
| Lightsage | Managed API, CLI, and MCP sandbox benchmarks with per-agent and per-endpoint results | Run a representative workflow, verify its resulting state, and confirm any SDK-specific requirements |
| Gauge CLI | Open-source local coding-agent evaluations and hosted options | Which inputs your chosen execution mode supports, and how it verifies the final outcome |
| Harbor | A benchmark defined through task instructions, environments, and verifiers | Work to build and maintain the task suite and execution setup |
| Inspect AI | Programmable agent evaluations and sandboxed agent integrations | Agent integration, model routing, configuration, and scoring |
| MCPJam | MCP inspection, OAuth debugging, and tool selection checks for CI | Which client behavior is exercised and how to verify task completion separately |
| Promptfoo | Coding-agent providers and configurable regression tests | The selected runtime, workspace permissions, MCP execution, and independent outcome assertions |
| MCP Inspector | MCP connections, capability inspection, and direct tool calls | Whether the failure is in the server or in the agent's use of it |
| LangSmith | Dataset evaluations, experiments, and production feedback | How the external agent and product environment enter the evaluation |
| Braintrust | Experiments, scorers, and evaluation of recorded or executed tasks | Which execution, environment, and outcome checks your setup supplies |
These categories describe useful entry points. Several tools cover more than one part of the workflow.
What an agent usability testing platform must demonstrate
Ask to see one complete product task before comparing a feature list:
- The actual agent runtime. Record the harness, model, effort, tools, permissions, and documentation available at the start.
- Independent verification. Read back the product state or execute the generated integration. Check required SDK, CLI, or MCP use separately from the final answer.
- An inspectable attempt. Follow the calls, errors, recovery, files, and outcome evidence. Review passes as well as failures.
- A repeatable comparison. Reset the starting state, repeat both conditions, and retain inconclusive and infrastructure-blocked attempts.
- An honest effort report. Separate token categories, elapsed task time, and tool latency where available. Attribute savings only after a controlled comparison.
The MCP task-success guide demonstrates this review on a passing Vercel task, including the failed request, recovery sequence, and measurement limits. For first-time installation and authentication, use the clean-project SDK test.
Managed platforms for product experiments
obss for agent usability and agent observability
obss combines custom product benchmarks with visibility into how agents perform the work. Define tasks and rubrics, then compare agents, models, and treatments such as skills, MCP servers, CLIs, and SDKs on managed cloud infrastructure.
Use Matrix to compare outcomes and resource use, Frictions to find recurring obstacles, and Traces and Output to inspect tool calls, commands, files, and delivered work. The Evals documentation explains requirement verdicts; Frictions documentation explains how to review recurring issues and their supporting trajectories. Review ambiguous verdicts because model-based grading can be wrong.
Choose obss as the main workflow when your team needs to measure agent usability, diagnose the behavior behind a result, and rerun after a product change. An SDK documentation revision, a new skill, or an MCP schema change can become a controlled comparison.
Lightsage
Lightsage’s Agent Usability page describes managed sandbox runs for API, CLI, and MCP workflows, with results broken out by agent and endpoint. Its public Devtool Arena lets buyers inspect how it presents comparative results. Its tools guide uses named agent coverage as its ranking criterion. That is one dated vendor methodology; it does not show which platform best verifies your task. Ask for a representative workflow, including your SDK if it matters, and inspect the resulting artifact or state. See Obss vs Lightsage for a closer comparison.
The Prompting Company
The Prompting Company's Agent Experience documentation describes real Claude Code, Codex, and OpenCode runs in fresh sandboxes, user-defined task goals, transcripts, sandbox artifacts, token usage, cost, and duration. Its workflow also groups failures and proposes fixes. For a buying decision, compare the same task, environment, acceptance checks, and evidence review with Obss; see our Obss vs The Prompting Company comparison.
Local or hosted evaluations with Gauge
The open-source Gauge CLI documents how to define a task and criteria, run a local benchmark using your agent credentials, then compare batches. Its README separates local inputs such as skills, npm CLIs and SDKs, docs, and web fixtures from additional hosted inputs such as authenticated APIs and MCP servers. Confirm that the execution mode you plan to use supports your fixture and credentials. Judge the resulting artifact or product state with a check separate from the agent’s answer.
Frameworks for a benchmark you operate
ax-eval and Microsoft Scope
ax-eval is an open-source option for reviewed tasks across API, CLI, SDK, and MCP surfaces, with harness execution and independent live-state verification. The team operating it owns the product-specific task pack, credential and sandbox contract, and verification/reset rules. It is a concrete starting point when you want to own the runner and test definitions.
Microsoft Scope is an open-source Agent Experience evaluation platform for repeatable tasks, reusable criteria, and inspectable run evidence. Its documentation reviewed November 2026 lists GitHub Copilot and Claude Code as supported, with Codex and OpenCode planned. Confirm current integration and hosting requirements for the actual pilot; do not assume every named agent is already executable.
Harbor represents tasks as instructions, an environment, and a verification script. It suits teams that want to define executable success checks and own the benchmark implementation. Compare the work of creating fixtures, integrating agents, running trials, and maintaining the suite.
Inspect AI provides programmable evaluation infrastructure. Its Agent Bridge supports integrations with external agents, including CLI agents running in sandboxes. The bridge routes model calls through Inspect, so review model routing and generation settings when reproducing a customer configuration.
A framework is a useful choice when flexibility and integration control justify the operating work. Include that work in the comparison with a managed platform.
MCP testing and diagnostic tools
MCPJam is a starting point for MCP-specific client behavior, tool-choice tests, and authorization debugging. The OAuth debugger helps inspect the authentication flow. Its testing SDK lets you put tool behavior assertions in code. Confirm which clients or client configurations are exercised, then add an independent check when success depends on the resulting artifact or product state. See Obss vs MCPJam for how these layers fit together.
Promptfoo's coding-agent guide documents Codex SDK/app-server, Claude Agent SDK, and OpenCode SDK providers. Select the runtime boundary your customers use, configure the workspace and permissions, and add independent checks of the result. Its direct MCP provider invokes tools directly; that alone does not test whether an autonomous agent discovers and chooses the tool. A model or coding-agent run with MCP access answers a different question.
MCP Inspector helps isolate connection and server problems through capability inspection and direct calls. Use it to confirm that a known-good operation works with the intended permissions. Follow that check with an agent task to evaluate discovery, argument selection, and completion.
Evaluation and production observability stacks
LangSmith documents offline and online evaluations, datasets, evaluators, and experiment comparisons. Braintrust documents datasets, custom tasks, scorers, experiments, and production scoring, including workflows for more complex agent execution.
For production agent observability, establish how the system collects application traces and what you must instrument. For a product usability experiment, also establish who supplies the external agent runner, environment, reset logic, and outcome checks. obss supplies the controlled experiment and its recorded execution evidence; compare that complete workflow with extending a stack you already operate.
A practical pilot for choosing a tool
Use three tasks:
Install and authenticate from a clean project.
Complete an integration inside an existing application.
Recover from an expected error and finish the intended operation.
Test the current product setup against one proposed improvement, such as revised authentication instructions. Keep the product version, permissions, starting data, and agent configuration fixed. Reset the environment before each attempt.
Verify the outcome across interfaces
For example, ask an agent to export all matching records for one tenant from a fixture with more than one page of data. Use the same record IDs as ground truth for all supported interfaces. For an API, retain the requests and cursor history; for an SDK, run the generated integration in a clean project; for a CLI, retain the command, exit status, and output file; for MCP, record the selected tools and arguments. In every condition, compare the exported IDs with the fixture, checking for missing, duplicate, or cross-tenant records. See how these checks were applied in the matched ClickHouse pilot.
For an initial diagnostic exercise, three tasks × two agent configurations × two setups × three trials produces 36 attempts. That is enough to inspect representative behavior; it does not establish a precise reliability estimate.
For each attempt, collect:
| Measure | What it tells you |
|---|---|
| Verified task outcome | Whether the requested state or working artifact exists |
| Required product or interface use | Whether the agent completed the work through the path being evaluated |
| Failure and recovery evidence | Where the attempt first became blocked and what happened next |
| Steps, retries, tokens, and time | What successful and failed attempts cost |
| Setup and maintenance effort | What your team must operate beyond the reported score |
Review failed attempts and sample the passes. In our vector-database trajectory review, an agent produced a correct result after abandoning the intended product and finishing with SQLite. A plausible answer and successful product use require separate checks.
When comparing costs, use the same tasks, trial count, and agent configurations. Include platform charges, model inference, infrastructure, integration, and maintenance. For example, obss's platform pricing separates run charges from model inference.
Our 20-task harness study illustrates why outcome and resource use belong together. With the same Opus 5 high-effort configuration, OpenCode and Claude Code both reached a 67.5% pass rate, while Claude Code used roughly twice the tokens. Those findings describe that task set and configuration.
Common questions
What does agent observability mean in obss?
It means inspecting recorded experiment executions through traces, output, metrics, and friction findings. You can review tool calls and commands, see what the agent read or changed, and connect those actions to a rubric verdict. Continuous production trace collection is a separate integration requirement to establish when selecting an observability stack.
Can I test Agent Experience with MCP Inspector alone?
Inspector can establish that a server connects and its operations work. To evaluate usability, also give an agent a customer task and verify the result. That introduces discovery, tool selection, sequencing, and recovery into the test.
How is product Agent Experience different from evaluating an agent we built?
The test target changes. A product AX evaluation asks how external agents use your software. It needs the interfaces, documentation, permissions, and environment those agents encounter. Your existing evaluation stack can still supply parts of that system.
Should we build a harness or buy a platform?
Start by implementing one task in each approach. Compare evidence quality, execution fit, recurring cost, and maintenance. Keep the task definitions and acceptance criteria understandable to your team whichever tool operates the runs.
How do we know an agent actually used our SDK or MCP server?
Make use of the required interface an explicit acceptance criterion. Inspect the execution evidence and check the resulting product state. Track unrestricted interface choice as a separate experiment.
Start with a workflow you need to improve
Bring a failing integration, a documentation revision, or an interface change you are considering. Use obss to measure agent usability, observe the execution, identify friction, and compare the next version against the current setup.
, or book an obss demo to turn your own workflow into an experiment.
By the Obss team. Updated November 2026. Tool documentation was checked on October 26; experiment counts retain the snapshot versions and review dates stated above. Vendor capabilities describe their documentation, not a head-to-head performance result.
For a controlled next experiment, evaluate SKILL.md activation, task success, tokens, and latency or compare CLI, MCP, and SDK setups on matched customer tasks. Both guides include real RunType results and reusable evaluation plans.
Related resources
- How to test MCP task success beyond protocol checks
- How to test SDK usability with Claude Code and Codex
- Agent observability: find why coding agents fail to use your product
- Build or buy an agent usability test harness?
- obss vs The Prompting Company for agent usability and observability
- obss vs 2027.dev (Firegoat) for agent usability and observability
- obss vs Armature for agent usability and observability
- obss vs MCPJam for agent usability and observability