Skip to main content
All resources

By obss team. Updated . Experiment figures identify their source snapshots and review dates below.

How to test MCP task success beyond protocol checks

To test whether agents can actually use your MCP server, give them a realistic customer task, start each attempt from a known state, and independently verify the result in your product. Record the tool calls and recovery steps so you can explain failures. Repeat the same task before and after a bounded change.

For the managed product workflow, see agent usability testing with Obss: define customer tasks, compare interface conditions, and inspect the execution evidence behind the outcome.

obss is an agent usability testing and agent observability platform for this workflow. It organizes controlled experiments around tasks, interface treatments, agent configurations, and repeated trials, then connects requirement-level results with recorded execution evidence. Its experiment model keeps the task's files and machine fixed while treatments change the available tools or instructions.

What protocol checks establish

A valid response to a tool call is useful evidence. It does not establish that an agent will discover the tool, choose it at the right time, supply appropriate inputs, or complete a multi-step workflow.

Use MCP Inspector to inspect your server and make explicit calls. Its browser, CLI, and terminal interfaces support protocol debugging. Add an agent-driven task suite to test behavior when the agent has to choose its own actions.

Layer Question Evidence to retain
Connection and protocol Can a correctly configured client connect and call the tool? Initialization, tool schemas, explicit requests and responses
Agent interaction Can the target agent discover, select, and use the tool? Available tools, chosen calls, arguments, errors, recovery
Workflow outcome Did the intended product operation actually happen? Persisted state, returned identifiers, executable acceptance checks
Evidence quality Does the final answer follow from what the agent observed? Relevant tool results and the claims they support

Keep the layers separate in the report. A connection failure needs a different fix from a well-connected agent making an unsupported diagnosis.

A passing task can still expose an MCP usability problem

An agent that eventually succeeds may still spend calls and reasoning recovering from an unclear interface. Record that path alongside the pass/fail verdict. In the case below, the invalid input caused the request to fail; the design question is whether the tool description and error helped the agent find the supported path.

Vercel MCP case study: the file-ID handoff

We tested Vercel's MCP server at https://mcp.vercel.com. One task asked the agent to diagnose a failed build. The agent needed to inspect the deployed build script, connect it to the failure evidence, and report the diagnosis. This required discovering the deployment file and retrieving its contents through the MCP.

The historical attempt ran on September 24, 2026 with Claude Code and Opus 5 at high effort. In the public Vercel experiment, snapshot v4 reviewed October 19, locate run r11, trial31, and trajectory steps 21, 23, and 25. The task ultimately passed.

Step Observed interaction What it establishes
21 get_deployment_file_contents received path: "scripts/build.cjs" and placeholder fileId: "x" The request contained an invalid file ID
21 404 Not Found, code not_found, message File not found: x The error identified the rejected value but did not explain how to obtain a valid ID
23 The agent called list_deployment_files and obtained the file entry's uid The listing supplied the identifier needed for the read
25 The agent passed that uid as get_deployment_file_contents.fileId and retrieved the contents A valid file ID worked; the agent recovered and completed the task

The handoff is list_deployment_files → selected file entry's uid → get_deployment_file_contents.fileId. Vercel's current deployment-tool reference marks fileId as required and limits path to Git deployments. This attempt does not establish a dead endpoint or a failed read for valid input. It exposes a discoverability and recovery question: can an agent infer the identifier mapping without guessing?

What a more useful error could say

The following is a proposed message, not a response Vercel returned:

File ID "x" was not found for this deployment. Call list_deployment_files for this deployment, locate scripts/build.cjs, and pass that entry's uid as fileId. The path parameter is available only for Git deployments.

A description or error like this names the prerequisite operation, the returned field, and the parameter that consumes it. The next test is whether that bounded change reduces invalid reads under the same task and starting conditions.

How many calls, tokens, and seconds did recovery involve?

Measurement Observed value Interpretation
Lookup calls from the failed read through the successful read 3 MCP calls One failed read plus the valid list and read: one extra failed call
Model turns after the failed-request turn 2 turns Turns 8 and 9 also included a legitimate source-hash check
Output tokens in those two turns 995 Generated tokens for the recovery-and-verification sequence
Input tokens in those two turns 139,739 total: 136,015 cached and 3,724 uncached Input-context usage; keep cached and uncached categories separate
Elapsed recovery interval 23.319 seconds From the start of turn 8 to the start of turn 10, after the successful read; not isolated API latency

Only the additional failed call is established as extra in this inspected sequence. The 995 output tokens and 23.319 seconds include work needed to discover, read, and verify the file; they are not proven savings from a better message. Usage was available per complete model turn, and individual tool durations were unavailable. This is one historical passing attempt, not an average friction cost. Establish avoidable tokens and time with repeated original-versus-revised trials.

For the broader workflow, agent observability connects this interaction history to the task verdict. The evaluation-tools guide explains what to require from a platform that captures both.

A real example: a strong aggregate hid a weak workflow

obss's public Vercel MCP experiment contains 60 attempts: ten tasks, two agent configurations, and three trials per combination. In published snapshot v4, reviewed October 19, 2026, Claude Code 2.1.233 with Opus 5 at high effort passed 28 of 30; Codex 0.147.0 with gpt-5.6-sol at medium effort also passed 28 of 30. That is 56 of 60 overall.

One workflow tells a more specific story. Task 10 concerned an incident with insufficient evidence. Only three of six attempts passed. The failed requirement concerned choosing a next diagnostic step that would obtain evidence capable of distinguishing plausible historical upstream causes.

Successful access and many successful tasks did not establish that this diagnostic workflow worked reliably. The actionable result was the task-level evidence gap, which an aggregate pass rate could obscure. These findings describe the configured test projects, integration, tasks, and agent versions; they are not a general failure-rate claim about Vercel.

Build a small suite around customer outcomes

Start with workflows that require different kinds of work: initial setup, a normal operation involving several tools, and recovery or diagnosis. Write a goal a customer would recognize. For example:

In the supplied test project, create a deployment, verify that the expected endpoint serves the fixture data, and report the deployment identifier and verification result.

This is an illustrative task, not a claim about a completed experiment. Its rubric should check the created deployment, the actual endpoint response, and the accuracy of the reported identifier. If MCP usage is the purpose of the test, check evidence of that path too.

Decide what fallback means before running the suite. A task completed through a raw HTTP request may satisfy the customer's broader goal while failing to demonstrate MCP usability. Record both results. Do not quietly mark product abandonment as a successful MCP integration. Our trajectory analysis shows why product participation needs its own check.

Control setup without hiding onboarding failures

Run two tracks when setup matters. A clean-start track includes installation, configuration, and authentication using the materials customers receive. A configured track supplies working access so you can isolate operational usability. Label the tracks rather than combining them into an unexplained score.

For every attempt, record the product and MCP version, harness and model, effort setting, accessible documentation, credentials' scope, starting fixtures, and execution budget. Use separate resettable test state for parallel attempts. Keep expected answers and grading credentials outside the agent's workspace.

When testing a tool-description change, keep the rest of the setup fixed. Giving one condition extra examples or broader permissions changes the question the experiment answers.

Diagnose the first broken transition

For a failed attempt, identify the last verified success and the next transition that did not work. Then inspect its evidence.

Observed failure Next check
Server never became available Transport, configuration, authentication, and environment setup
Tool was available but never selected Discoverability, names, descriptions, and competing paths
Repeated invalid arguments Schema clarity, examples, prerequisites, and error feedback
Tool calls succeeded but the task failed Workflow sequencing, persisted state, and acceptance criteria
Final answer exceeded the evidence Verification step and claim-to-source consistency

Read failures and a sample of passes. Some successful attempts contain expensive detours. obss's Traces retain commands, tool calls, results, files, and token use; Frictions analyze saved traces and output separately from rubric scoring.

Reusable MCP task-success checklist

Use this checklist for one customer workflow before adding a larger suite:

  • Task: state the customer outcome without revealing the expected tool sequence to the agent.
  • Starting state: version the project, fixtures, MCP server, credentials' scope, agent/model, effort, and documentation access.
  • Outcome check: independently read back the expected state or verify the delivered artifact and diagnosis.
  • Product-use check: require evidence of MCP participation when MCP usability is the test target; record fallbacks separately.
  • Interaction review: inspect wrong identifiers, missing prerequisites, repeated errors, and recovery in failed attempts and a sample of passes.
  • Metrics: retain task success, invalid calls, retries, output tokens, cached/uncached input tokens, and clearly defined elapsed intervals.
  • Comparison: change one description or error response, reset the starting state, and repeat both conditions under the same budgets and rubric.

For the file-ID case, count placeholder/path-as-ID attempts, whether the agent obtains the correct uid, successful content retrieval, and completion of the original diagnosis. A lower invalid-call count without a correct diagnosis is not sufficient. Obss's task, treatment, agent, and experiment model supplies the structure; Traces and Insights let the team inspect the interaction behind each result.

Turn a diagnosis into a regression test

Change one suspected cause, rerun the same tasks under the same configurations, and report successful attempts over total attempts. Compare retries, elapsed time, and tokens alongside outcomes. Keep infrastructure-blocked and inconclusive attempts visible rather than counting them as product failures.

A few repeats can expose an obvious problem. They do not establish a rare-failure guarantee. Retain the task, rubric, fixture, version records, and representative traces so the next API or MCP release can be checked against the same workflow.

Questions teams ask

Does passing MCP Inspector mean an agent can use our server?

It establishes behavior under the calls you tested. Real-task usability also depends on discovery, selection, sequencing, recovery, and outcome verification in the target agent environment.

What should count as success?

Use independently verified product state and the task's requirements. Add a separate product-use check when the interface matters. A final message saying the task is done is insufficient evidence on its own.

Where should we start with obss?

Bring one failing customer workflow and one suspected obstacle. Define the task and rubric, compare the current interface with a bounded revision, and inspect the evidence behind the result. See the agent usability and observability tools guide for selection criteria or obss vs Lightsage for a product-focused comparison.

For a controlled next experiment, evaluate SKILL.md activation, task success, tokens, and latency or compare CLI, MCP, and SDK setups on matched customer tasks. Both guides include real RunType results and reusable evaluation plans.

Related resources

Book an obss demo with one workflow, its current setup, and the change you want to evaluate.