Claude Agent Observability: Trace Model Calls, Tools, Cost, and Failures

The short answer

An agent that returns an answer is not necessarily healthy. Production teams need to know which request started a run, which model calls and tools followed, where time and cost accumulated, what failed, and whether the user task succeeded.

This guide shows how to use an OpenTelemetry-shaped operating model for a Claude agent without turning traces into a second data leak.

Start with a trace that follows the user task

Create one parent trace for the application request. Keep the trace context through the agent run, model calls, tool calls, retries, queues, and the final application response. Anthropic's Agent SDK materials describe native OpenTelemetry metrics and events when telemetry is enabled; OpenTelemetry provides vendor-neutral GenAI conventions for agent, model, and tool operations.

Use a small span model

Keep content out by default

Do not put raw prompts, retrieved documents, secrets, tokens, or customer data into ordinary telemetry. Use hashed or redacted identifiers, explicit opt-in sampling, access controls, and short retention for diagnostic payloads. Record metadata that answers an operational question. If a field does not help diagnose reliability, cost, safety, or outcome, it is probably not needed.

Turn spans into operating signals

Track p50 and p95 end-to-end latency, model latency, tool latency, token use, cost per completed task, tool error rate, retry rate, policy blocks, human escalation rate, and task-success rate. A low model-error rate can coexist with a poor user outcome if a slow or incorrect tool causes the final failure.

Design alerts around decisions

Alert when an owner can act. Examples: tool failures exceed a threshold, cost per completed task rises sharply, a new prompt version reduces evaluation results, or an approval queue grows beyond its service target. Attach the runbook link, owner, query, and first containment action to the alert.

Make evaluation and telemetry meet

Use the same request and version identifiers in offline evaluations, canary releases, and production traces. Then a regression can be traced to a model change, tool schema, retrieval version, policy, or deployment rather than blamed on the agent in general.

A practical operating checklist

• Trace parent application requests through model and tool work.
• Record model, tool, cost, latency, retry, and outcome metadata.
• Redact content and restrict access to diagnostic payloads.
• Version prompts, tools, policies, retrieval, and evaluation datasets.
• Define a task-success metric and human-review signal.
• Create alerts only for conditions with a named owner and response.
• Test a failure and confirm the trace leads to a usable investigation.

Where ClaudeExperts fits

ClaudeExperts builds and audits production Claude systems with evaluation harnesses, controlled tool boundaries, and operational ownership. Observability is not a dashboard add-on. It is the evidence layer that makes cost, quality, and failure behavior reviewable after the agent leaves a happy-path demo.

Sources

• Anthropic: Agent SDK observability
• OpenTelemetry: GenAI observability
• OpenTelemetry GenAI semantic conventions