Why most Claude pilots never reach production

A surprising number of Claude pilots look healthy in week two and are still a demo in month nine. The prompt is rarely the reason. The system around the model is.

This is the pattern we see when we are asked to audit a stalled build. Use it as a checklist before you spend another quarter on a prototype that cannot take traffic.

The prompt was never the bottleneck

Teams over-invest in prompt wording because it is visible and cheap to iterate. Production asks different questions:

• Can the model call the right tools, with the right permissions, on the first attempt?
• What happens when the tool fails, times out, or returns a partial payload?
• Who reviews a high-impact action before it lands in Salesforce, SAP or the claims system?
• How do you know last week's prompt change made quality worse?

If you cannot answer those, you do not have a Claude system. You have a chat window with credentials nearby.

Four failure modes that keep pilots in staging

1. Tools designed like API wrappers

Dumping an internal API into a tool schema and hoping Claude will "figure it out" produces retries, missing fields and silent wrong writes. Tool design is product work: narrow actions, explicit required fields, human-readable errors, and a hard boundary around what the model is allowed to mutate.

This is why MCP and tool design shows up in almost every recovery engagement. The protocol is not the point. The contract Claude has with your systems of record is.

2. No evaluation, so nobody can ship

Without a golden set and a judge rubric, every model or prompt change becomes a meeting. Teams freeze the prompt. They stop touching retrieval. They never turn the feature on for more than a handful of friendly users.

An eval harness does not need to be elaborate. It needs to be in CI, run on every change, and report a number a VP can read. We wrote up the minimum setup in how to put eval gates on a Claude agent and the CI shape in eval gates in CI.

3. Cost and latency treated as afterthoughts

Uncached long contexts, oversized retrieval windows and unbounded agent loops make a demo look magical and a production bill look alarming. The typical overspend we measure on inherited agent systems is 3.4x, usually from context that did not need to be there.

Prompt caching, structured outputs and a context budget belong in week one, not after finance notices the invoice.

4. No safety path that security will sign

Injection resistance, PII handling, audit logs and a human review queue are not polish. In healthcare, financial services and legal work they are the release gate. A pilot that cannot survive a security review will never see production traffic, no matter how good the answers look in Slack.

What "production" actually means here

For our work, a Claude system is in production when all of this is true:

• It serves real users or real operators, not a staging tenant.
• Tool calls hit live systems of record, scoped and logged.
• Quality is measured on a held-out set every time the prompt, tools or retrieval change.
• Spend and latency have ceilings, not vibes.
• There is a documented path for refusal, retry and human takeover.

That is a six-week job when the scope is one outcome, not a platform rewrite. See the production agent build if that is the shape of work you need.

A two-week way to find the real blocker

If you already have a prototype, do not start by rewriting the prompt. Spend two weeks measuring:

1. Prompt and tool review against live traces.
2. Cost and latency baseline on representative traffic.
3. Injection and data-handling review.
4. A ranked remediation plan with numbers you can retest.

That is the Claude Expert Audit. The output is not a slide deck of best practices. It is a list of what is broken, in order, with evidence.

Ship the seam, not the demo

The teams that get unstuck pick one production seam, put evals and guardrails on it, and take that slice live. Everything else waits. Breadth is how pilots age. Depth is how they graduate.

If you want a specialist inside the repository while that happens, book a 15-minute discussion.