Claude latency in production: timeouts, streaming and budgets
Latency is a budget. Cap steps, cache the prefix, stream when a human is waiting, and fail before the user does.
Latency for a Claude agent is not one model round-trip. It is the sum of retrieval, each tool, each model call and your own queue. Users feel the sum. Your SLOs should too.
Unbounded loops are how a six-second demo becomes a 90-second production stall. Put ceilings on steps and wall-clock time in agent architecture, then tune.
Levers that actually move p95
- Cache the stable prefix. See prompt caching.
- Retrieve less. See retrieval.
- Fewer tools per turn. Parallelise only what is safe.
- Stream tokens when a human is staring at a UI. Do not stream a backend job that will be applied as JSON.
- Fail closed on timeout. A partial write is worse than a retry.
Measure task time, not token time
A fast wrong answer is still a failure. Pair latency dashboards with eval pass rate. Optimising only for speed will strip the context the model needed.
Timeouts are product copy
Tell the user the agent is still working, or that it queued a human, or that it failed. Silence is how people click twice and duplicate a write.
If p95 is the reason you cannot turn the feature on, it will show up in an audit next to spend. They are usually the same bloated prompt.
Related notes
Quality and cost
Claude Agent Observability: Trace Model Calls, Tools, Cost, and Failures
A production guide to tracing Claude model calls, tools, cost, failures, and task outcomes with OpenTelemetry-shaped telemetry.
Quality and cost
Put eval gates on your Claude agent before you scale traffic
An eval harness turns Claude quality into a CI number. Here is the minimum setup that catches regressions before customers do.
Quality and cost
Prompt caching is how most Claude bills actually come down
Uncached long context is the usual 3x overspend. Cache the stable prefix, budget the rest, and measure cost per successful task.