By obss team. Updated . Experiment figures identify their source snapshots and review dates below.
How to test MCP task success beyond protocol checks
To test whether agents can actually use your MCP server, give them a realistic customer task, start each attempt from a known state, and independently verify the result in your product. Record the tool calls and recovery steps so you can explain failures. Repeat the same task before and after a bounded change.
For the managed product workflow, see agent usability testing with Obss: define customer tasks, compare interface conditions, and inspect the execution evidence behind the outcome.
obss is an agent usability testing and agent observability platform for this workflow. It organizes controlled experiments around tasks, interface treatments, agent configurations, and repeated trials, then connects requirement-level results with recorded execution evidence. Its experiment model keeps the task's files and machine fixed while treatments change the available tools or instructions.
What protocol checks establish
A valid response to a tool call is useful evidence. It does not establish that an agent will discover the tool, choose it at the right time, supply appropriate inputs, or complete a multi-step workflow.
Use MCP Inspector to inspect your server and make explicit calls. Its browser, CLI, and terminal interfaces support protocol debugging. Add an agent-driven task suite to test behavior when the agent has to choose its own actions.
| Layer | Question | Evidence to retain |
|---|---|---|
| Connection and protocol | Can a correctly configured client connect and call the tool? | Initialization, tool schemas, explicit requests and responses |
| Agent interaction | Can the target agent discover, select, and use the tool? | Available tools, chosen calls, arguments, errors, recovery |
| Workflow outcome | Did the intended product operation actually happen? | Persisted state, returned identifiers, executable acceptance checks |
| Evidence quality | Does the final answer follow from what the agent observed? | Relevant tool results and the claims they support |
Keep the layers separate in the report. A connection failure needs a different fix from a well-connected agent making an unsupported diagnosis.
A passing task can still expose an MCP usability problem
An agent that eventually succeeds may still spend calls and reasoning recovering from an unclear interface. Record that path alongside the pass/fail verdict. In the case below, the invalid input caused the request to fail; the design question is whether the tool description and error helped the agent find the supported path.
Vercel MCP case study: the file-ID handoff
We tested Vercel's MCP server at https://mcp.vercel.com. One task asked the agent to diagnose a failed build. The agent needed to inspect the deployed build script, connect it to the failure evidence, and report the diagnosis. This required discovering the deployment file and retrieving its contents through the MCP.
The historical attempt ran on September 24, 2026 with Claude Code and Opus 5 at high effort. In the public Vercel experiment, snapshot v4 reviewed October 19, locate run r11, trial31, and trajectory steps 21, 23, and 25. The task ultimately passed.
| Step | Observed interaction | What it establishes |
|---|---|---|
| 21 | get_deployment_file_contents received path: "scripts/build.cjs" and placeholder fileId: "x" |
The request contained an invalid file ID |
| 21 | 404 Not Found, code not_found, message File not found: x |
The error identified the rejected value but did not explain how to obtain a valid ID |
| 23 | The agent called list_deployment_files and obtained the file entry's uid |
The listing supplied the identifier needed for the read |
| 25 | The agent passed that uid as get_deployment_file_contents.fileId and retrieved the contents |
A valid file ID worked; the agent recovered and completed the task |
The handoff is list_deployment_files → selected file entry's uid → get_deployment_file_contents.fileId. Vercel's current deployment-tool reference marks fileId as required and limits path to Git deployments. This attempt does not establish a dead endpoint or a failed read for valid input. It exposes a discoverability and recovery question: can an agent infer the identifier mapping without guessing?
What a more useful error could say
The following is a proposed message, not a response Vercel returned:
File ID "x" was not found for this deployment. Call list_deployment_files for this deployment, locate scripts/build.cjs, and pass that entry's uid as fileId. The path parameter is available only for Git deployments.
A description or error like this names the prerequisite operation, the returned field, and the parameter that consumes it. The next test is whether that bounded change reduces invalid reads under the same task and starting conditions.
How many calls, tokens, and seconds did recovery involve?
| Measurement | Observed value | Interpretation |
|---|---|---|
| Lookup calls from the failed read through the successful read | 3 MCP calls | One failed read plus the valid list and read: one extra failed call |
| Model turns after the failed-request turn | 2 turns | Turns 8 and 9 also included a legitimate source-hash check |
| Output tokens in those two turns | 995 | Generated tokens for the recovery-and-verification sequence |
| Input tokens in those two turns | 139,739 total: 136,015 cached and 3,724 uncached | Input-context usage; keep cached and uncached categories separate |
| Elapsed recovery interval | 23.319 seconds | From the start of turn 8 to the start of turn 10, after the successful read; not isolated API latency |
Only the additional failed call is established as extra in this inspected sequence. The 995 output tokens and 23.319 seconds include work needed to discover, read, and verify the file; they are not proven savings from a better message. Usage was available per complete model turn, and individual tool durations were unavailable. This is one historical passing attempt, not an average friction cost. Establish avoidable tokens and time with repeated original-versus-revised trials.
For the broader workflow, agent observability connects this interaction history to the task verdict. The evaluation-tools guide explains what to require from a platform that captures both.
A real example: a strong aggregate hid a weak workflow
obss's public Vercel MCP experiment contains 60 attempts: ten tasks, two agent configurations, and three trials per combination. In published snapshot v4, reviewed October 19, 2026, Claude Code 2.1.233 with Opus 5 at high effort passed 28 of 30; Codex 0.147.0 with gpt-5.6-sol at medium effort also passed 28 of 30. That is 56 of 60 overall.
One workflow tells a more specific story. Task 10 concerned an incident with insufficient evidence. Only three of six attempts passed. The failed requirement concerned choosing a next diagnostic step that would obtain evidence capable of distinguishing plausible historical upstream causes.
Successful access and many successful tasks did not establish that this diagnostic workflow worked reliably. The actionable result was the task-level evidence gap, which an aggregate pass rate could obscure. These findings describe the configured test projects, integration, tasks, and agent versions; they are not a general failure-rate claim about Vercel.
Build a small suite around customer outcomes
Start with workflows that require different kinds of work: initial setup, a normal operation involving several tools, and recovery or diagnosis. Write a goal a customer would recognize. For example:
In the supplied test project, create a deployment, verify that the expected endpoint serves the fixture data, and report the deployment identifier and verification result.
This is an illustrative task, not a claim about a completed experiment. Its rubric should check the created deployment, the actual endpoint response, and the accuracy of the reported identifier. If MCP usage is the purpose of the test, check evidence of that path too.
Decide what fallback means before running the suite. A task completed through a raw HTTP request may satisfy the customer's broader goal while failing to demonstrate MCP usability. Record both results. Do not quietly mark product abandonment as a successful MCP integration. Our trajectory analysis shows why product participation needs its own check.
Control setup without hiding onboarding failures
Run two tracks when setup matters. A clean-start track includes installation, configuration, and authentication using the materials customers receive. A configured track supplies working access so you can isolate operational usability. Label the tracks rather than combining them into an unexplained score.
For every attempt, record the product and MCP version, harness and model, effort setting, accessible documentation, credentials' scope, starting fixtures, and execution budget. Use separate resettable test state for parallel attempts. Keep expected answers and grading credentials outside the agent's workspace.
When testing a tool-description change, keep the rest of the setup fixed. Giving one condition extra examples or broader permissions changes the question the experiment answers.
Diagnose the first broken transition
For a failed attempt, identify the last verified success and the next transition that did not work. Then inspect its evidence.
| Observed failure | Next check |
|---|---|
| Server never became available | Transport, configuration, authentication, and environment setup |
| Tool was available but never selected | Discoverability, names, descriptions, and competing paths |
| Repeated invalid arguments | Schema clarity, examples, prerequisites, and error feedback |
| Tool calls succeeded but the task failed | Workflow sequencing, persisted state, and acceptance criteria |
| Final answer exceeded the evidence | Verification step and claim-to-source consistency |
Read failures and a sample of passes. Some successful attempts contain expensive detours. obss's Traces retain commands, tool calls, results, files, and token use; Frictions analyze saved traces and output separately from rubric scoring.
Reusable MCP task-success checklist
Use this checklist for one customer workflow before adding a larger suite:
- Task: state the customer outcome without revealing the expected tool sequence to the agent.
- Starting state: version the project, fixtures, MCP server, credentials' scope, agent/model, effort, and documentation access.
- Outcome check: independently read back the expected state or verify the delivered artifact and diagnosis.
- Product-use check: require evidence of MCP participation when MCP usability is the test target; record fallbacks separately.
- Interaction review: inspect wrong identifiers, missing prerequisites, repeated errors, and recovery in failed attempts and a sample of passes.
- Metrics: retain task success, invalid calls, retries, output tokens, cached/uncached input tokens, and clearly defined elapsed intervals.
- Comparison: change one description or error response, reset the starting state, and repeat both conditions under the same budgets and rubric.
For the file-ID case, count placeholder/path-as-ID attempts, whether the agent obtains the correct uid, successful content retrieval, and completion of the original diagnosis. A lower invalid-call count without a correct diagnosis is not sufficient. Obss's task, treatment, agent, and experiment model supplies the structure; Traces and Insights let the team inspect the interaction behind each result.
Turn a diagnosis into a regression test
Change one suspected cause, rerun the same tasks under the same configurations, and report successful attempts over total attempts. Compare retries, elapsed time, and tokens alongside outcomes. Keep infrastructure-blocked and inconclusive attempts visible rather than counting them as product failures.
A few repeats can expose an obvious problem. They do not establish a rare-failure guarantee. Retain the task, rubric, fixture, version records, and representative traces so the next API or MCP release can be checked against the same workflow.
Questions teams ask
Does passing MCP Inspector mean an agent can use our server?
It establishes behavior under the calls you tested. Real-task usability also depends on discovery, selection, sequencing, recovery, and outcome verification in the target agent environment.
What should count as success?
Use independently verified product state and the task's requirements. Add a separate product-use check when the interface matters. A final message saying the task is done is insufficient evidence on its own.
Where should we start with obss?
Bring one failing customer workflow and one suspected obstacle. Define the task and rubric, compare the current interface with a bounded revision, and inspect the evidence behind the result. See the agent usability and observability tools guide for selection criteria or obss vs Lightsage for a product-focused comparison.
For a controlled next experiment, evaluate SKILL.md activation, task success, tokens, and latency or compare CLI, MCP, and SDK setups on matched customer tasks. Both guides include real RunType results and reusable evaluation plans.
Related resources
- How to test SDK usability with Claude Code and Codex
- Agent observability: find why coding agents fail to use your product
- Build or buy an agent usability test harness?
- obss vs The Prompting Company for agent usability and observability
- obss vs 2027.dev (Firegoat) for agent usability and observability
- obss vs Armature for agent usability and observability
- obss vs MCPJam for agent usability and observability
Book an obss demo with one workflow, its current setup, and the change you want to evaluate.