Skip to main content
All resources

By obss team. Updated . Experiment figures identify their source snapshots and review dates below.

Agent observability: find why coding agents fail to use your product

Agent observability records what an agent did, what each operation returned, and how that path relates to the customer's outcome. For product usability, connect trajectories with verified results, actual product use, errors, retries, tokens, and elapsed time. This reveals both failed workflows and detours hidden inside passing tasks.

For the managed product workflow, see agent usability testing with Obss: define customer tasks, compare interface conditions, and inspect the execution evidence behind the outcome.

Obss combines agent observability and agent usability testing in controlled product experiments. Run coding agents against your API, SDK, CLI, MCP server, docs, or skills; inspect the recorded attempts; then compare a bounded interface change. This guide concerns recorded experiment executions. Instrumenting every customer's live production session requires a separate telemetry setup.

Evidence review can change how you interpret a verdict

In our 32-attempt SDK documentation experiment, all 24 integration implementations pass their supplied checks on independent replay, while the original model grader flags one persistence attempt over cosine vector magnitude. The follow-up diagnostic found the same behavior in both conditions and a reference solution. Preserving the trajectory, exact submitted code, rubric and independent state checks made that disagreement inspectable. See how to inspect these runs in the Agent experience workflow, or browse the public research index for the underlying evidence.

Start with four kinds of evidence

Keep these signals linked but separately interpretable:

Evidence Question it answers What it cannot establish alone
Trace Which commands, calls, results, and file operations were recorded? Whether every business requirement was satisfied
Verified outcome Did the required product state or artifact exist? Whether the intended interface was used
Product participation Did the agent actually use the product or interface under test? Whether the user's whole task was completed correctly
Friction Where did the agent encounter errors, repeated work, or missing context? Whether the overall run failed

obss's Traces documentation describes ordered tool calls and results, commands, files read or changed, and token use. The Output view shows the final assistant message and recorded workspace changes; Evals score rubric requirements. These views support different questions about the same attempt.

Why a passing answer can hide product abandonment

In obss's study of coding-agent product navigation, researchers analyzed 112 baseline attempts across Claude Code and Codex. Seven used SQLite in place of the target vector database. The study also observed zero external documentation fetches or web searches across those attempts.

Those findings describe that task suite and setup. They do not establish that coding agents generally avoid documentation or that every SQLite solution was wrong for a user's broader goal. They show why product teams need evidence of the path taken.

If your evaluation is intended to measure a vector database integration, an output-only check can miss a substitute implementation. The trajectory analysis article develops this product-use problem. Add an explicit product-participation requirement when it is part of the decision, and retain the underlying task outcome as a separate result.

Missing evidence can look like a product failure

In our RunType ticket-triage experiment, each attempt had to create a saved flow or agent, run five tickets, and report their returned categories and priorities. In the SDK condition, both Codex trials passed the saved-object check but neither passed the combined execution-and-reporting requirement. Claude Code passed both requirements in both SDK trials. These are historical rubric outcomes from snapshot v3, with two repeats per configuration, rechecked November 2026.

Codex run r14, trial28's evaluation explains one failure: the captured execution output was truncated, so the evaluator could not verify every reported result against the returned values. It also found a separate priority error in the final report. The trace shows why a single failure label needs interpretation: an incomplete record and an incorrect business result call for different investigations.

Preserve complete returned values in a structured artifact, link them to fixture and operation IDs, and run an independent check. Treat missing evidence as its own diagnostic category; do not infer that an operation never happened or that the SDK caused truncation. The SDK usability case study contains the requirement-level table, configuration details, and links to the public trajectory.

Find the first failure in setup and authentication

When an agent gets stuck, reconstruct the transition from the last verified success to the first failed or unverified step. A useful onboarding sequence is discovery, installation, configuration, credential loading, authenticated operation, and outcome verification.

For example, an illustrative SDK run might install the right package, initialize a client, receive an authorization error, retry with the same configuration, and stop. The failed transition is the first authenticated operation. The authorization error alone does not prove whether the cause was missing permission, an incorrect account, expired access, or confusing instructions.

Retain enough evidence to investigate the alternatives:

  • The run, task, treatment, harness, model, and product versions.
  • The relevant command or tool call, its timestamp, response status, and sanitized error.
  • The credential source and permission scope, without exposing the secret value.
  • The selected account, project, and endpoint where these can be safely recorded.
  • The next action, whether it changed the suspected cause, and whether recovery succeeded.

Check environment preparation before assigning fault to the product. Missing test credentials, an unavailable service, and a broken fixture deserve explicit labels. They should remain visible without silently becoming evidence that the agent cannot use your interface.

Passing Vercel task: a file-ID handoff hid an extra failed call

We tested Vercel's MCP server with a task to diagnose a failed build. The agent needed to retrieve the deployed build script and connect it to the failure evidence. In run r11, trial31 of the public experiment, the task passed, but trajectory review found an invalid file read and recovery:

  1. Step 21 called get_deployment_file_contents with path: "scripts/build.cjs" and placeholder fileId: "x". It returned 404 and File not found: x.
  2. Step 23 called list_deployment_files and found the selected file entry's uid.
  3. Step 25 passed that uid as fileId and retrieved the contents.

The required handoff was list files → select the entry's uid → pass it as fileId. A valid ID worked. The usability question is whether the description and error make that prerequisite clear enough to prevent guessing. The MCP case study gives a proposed error that names the discovery tool and the returned field to use; that proposal has not been tested.

Recorded measure Value What it means
File-lookup sequence 3 MCP calls One failed read plus a valid listing and read: one extra failed call
Output tokens in recovery turns 8–9 995 Full-turn generation, including a legitimate source-hash check
Input tokens in those turns 139,739 total; 136,015 cached; 3,724 uncached Context usage, with cached and uncached input kept separate
Recovery interval 23.319 seconds Start of turn 8 to start of turn 10, after retrieval; not isolated API latency

This was one passing attempt from September 24, 2026, using Claude Code and Opus 5 at high effort, in public snapshot v4 reviewed October 19. Only the extra failed call is established as additional work. The token and time totals include required discovery, retrieval, and verification; they are not proven avoidable overhead, an average, or measured savings from a better error.

That is the difference observability makes to the next experiment: a passing verdict becomes a specific identifier-discovery question. Compare the original and revised message under the same task and configuration, count invalid-ID attempts, and verify that the diagnosis remains correct. Repeated controlled trials are needed to establish any token or latency improvement.

A minimum evidence record for agent observability

Evidence to retain Engineering decision it supports
Task, rubric, fixtures, product/interface version, agent harness, model, effort, and budgets Can this attempt be reproduced and compared?
Ordered calls or commands, timestamps, sanitized inputs, returned results, and next action Where did the first broken transition occur, and did recovery address it?
Complete returned business fields, artifact references, and independent read-back Was the outcome verified, or is the evidence incomplete?
Actual use of the required product/interface and any fallback Did the agent use the product, bypass it, or abandon it partway?
Retries, invalid calls, output tokens, cached/uncached input tokens, and defined time intervals What work accompanied the outcome, and what can be attributed to a change?

Keep credentials out of the record. Capture safe permission and environment context instead. Choose retention and export requirements for the environment you test; missing steps or truncated returns limit the conclusions even when the final report looks complete.

Use observability to test a product change

Turn a suspected cause into a bounded experiment. Keep the task, files, machine, permissions, and rubric fixed. Compare the current interface or instructions with a revised treatment under the same agent configurations and repeated trials.

Review the evidence in this order:

  1. Check verified task outcomes and product participation.
  2. Locate the relevant failure or detour in the trace.
  3. Inspect whether the revised condition changed that behavior.
  4. Compare retries, time, and token use alongside the outcome.
  5. Review a sample of passing attempts for hidden workarounds or unsupported claims.

obss's Frictions workflow analyzes saved traces and output through Insights, separately from rubric Evals. Teams can trigger Insights after a run and compare two or three runs side by side in Traces, Output, and Evals. A friction finding is a starting point for investigation; the rerun tests whether a proposed change helped.

What to ask of an observability tool

Ask whether it can move from a result to the relevant recorded interaction, show the actual responses behind a claim, preserve configuration details, and export the trajectory for review. Ask how it represents missing evidence and infrastructure failures. Verify retention and credential-handling requirements for the environment you intend to test.

For a product team, prioritize evidence that leads to a reproducible fix. An attractive score without task-level evidence is difficult to use in an engineering decision. The agent usability and observability tools guide provides a broader selection framework.

Questions teams ask

An agent finishes without using our product. How do we detect that?

Check execution evidence for the required product operation and verify the resulting state. Keep task correctness and product participation separate. Our product-bypass example shows why plausible output alone can miss a substitute implementation; a required-product task and an unbranded discovery task should be labeled separately.

How can we tell whether a fix reduced tokens or latency?

Repeat original and revised conditions with the same goal, fixtures, versions, budgets, and verification. Compare successful outcomes and product use first, then invalid calls, output tokens, cached/uncached input, and clearly defined elapsed intervals. A recovery-sequence total is not the same as avoidable cost.

How is agent observability related to agent usability testing?

Usability testing measures whether an agent can complete a representative product workflow under specified conditions. Observability supplies execution evidence that helps explain the outcome and select the next change to test. Use both to connect a result with an actionable diagnosis.

Does every friction mean the run failed?

No. An agent can recover and finish successfully. Keep friction, recovery, outcome, and resource use separate so the report shows both completion and the work required to get there.

Can obss show the exact cause of every failure?

Recorded evidence can narrow a failure to a specific interaction and support a diagnosis. Missing context or multiple plausible causes may require another controlled run. Start with one failing workflow in obss's experiment model and retain the evidence needed to test your explanation.

For a controlled next experiment, evaluate SKILL.md activation, task success, tokens, and latency or compare CLI, MCP, and SDK setups on matched customer tasks. Both guides include real RunType results and reusable evaluation plans.

Related resources

Book an obss demo with one workflow, its current setup, and the change you want to evaluate.