Put eval gates on your Claude agent before you scale traffic

Most teams shipping Claude still decide quality in a meeting. Someone pastes two answers into Slack. Someone else has a feeling. The change ships anyway.

That works until the agent is allowed to touch a real system. Then a quiet regression in retrieval, a tool schema tweak, or a model default you did not notice will land on a customer before anyone has language for what broke.

An eval harness is how you stop arguing and start measuring. It does not need a research team. It needs a golden set, a rubric, a gate in CI, and a dashboard that makes a drop obvious the day it happens.

What an eval harness is for

The job is not "score the model." The job is protect the behaviour you already accepted.

When we put a harness on a production agent, we are trying to answer four questions on every change:

• Did answer quality move, on the cases we care about?
• Did tool choice get worse: wrong tool, missing required fields, extra writes?
• Did we break a refusal, a PII rule, or a human-review trigger?
• Did cost or latency jump enough that the feature is no longer viable?

If you cannot answer those in CI, you are not ready to scale traffic. You are hoping.

The minimum setup that actually ships

Skip the 40-metric wish list. Start with this.

1. A golden set taken from real work

Fifty to two hundred cases is enough for a first gate if they are honest. Mine them from production traces, not from prompts you wrote to look good.

Include:

• Happy paths the agent must still complete.
• Tool-failure paths (timeouts, empty records, partial payloads).
• Borderline requests that should refuse or escalate.
• A handful of injection and data-exfil attempts.

Label the expected action, not just the expected prose. "Looks helpful" is not a label. "Create the Jira ticket with these fields, or ask a clarifying question" is.

2. A rubric a second human can apply

A judge, model or human, needs pass/fail criteria that do not require the original author in the room. We keep rubrics short:

• Correctness against source systems or the provided context.
• Tool discipline: allowed tools only, required fields present.
• Safety: no prohibited data, no skipped review step.
• Format: structured output that downstream code can parse.

Score each dimension. A single blended "quality" number hides the failure mode you need to fix.

3. The gate lives in CI, not in a notebook

If the suite is a Colab someone runs on Fridays, it will not run the week you are late. Wire it to pull request checks the same way you wire tests.

A useful default: block merge when a critical slice drops, and warn when a softer slice moves. Critical slices are safety, writes to systems of record, and the top customer journeys. Softer slices are tone and optional citations.

4. A dashboard leadership can read

Engineers will live in traces. Leadership needs a weekly view: pass rate by slice, cost per successful task, and the last change that moved either number. That is what turns model quality from a debate into an operating metric.

On our builds this is in the repo from week one, not as a phase-two platform project. See the eval and guardrail harness engagement if you want that installed on an agent you already have.

What we measure on inherited agents

When a team hands us a Claude agent with no suite, the first week is usually archaeological:

1. Reconstruct the actual user journeys from logs.
2. Freeze a snapshot of current behaviour so we do not "improve" it into something worse.
3. Add regression cases for the failures customers already hit.
4. Only then change prompts, tools or retrieval.

This sequence is why so many Claude pilots stall. People keep editing the prompt because they have no other feedback loop. The CI version of the gate is in eval gates in CI.

A note on model judges

LLM-as-judge is fine for language quality and rubric dimensions that are fuzzy. It is not fine as the only check on tool correctness or policy. Pair it with deterministic assertions:

• Did the tool name match the allow-list?
• Did the JSON parse?
• Did the write payload include the required keys?
• Did a PII pattern leak in the response?

The boring checks catch the expensive bugs. The judge catches the ones that look fluent and still wrong.

When to put this in before you build more

If the agent can send an email, update a record, or influence a customer decision, the harness comes before the next feature. New tools without a golden set just multiply the surface you cannot test.

If you are still in a prototype with no external side effects, a lightweight set of twenty cases is enough. Grow it the first time a human will rely on the output.

What you should leave with

A working harness is not a PDF of metrics. It is:

• A versioned dataset in the repository.
• A runner that executes on every pull request.
• A recorded baseline you can retest after we leave.
• A short rubric your team actually uses.

That is also what we hand over at the end of a production agent build. The agent is not done when it answers well once. It is done when a bad change is obvious by the next morning.

If you want that loop on your current Claude system, book a 15-minute discussion.