Put Claude eval gates in CI or accept shipping on instinct
If the golden set is not a pull request check, it will not run the week you are late. Block on safety and writes; warn on softer slices.
An eval gate is a check that fails a pull request when Claude quality, safety or tool behaviour drops. Notebooks are research. Gates are how you ship. The longer version of the design lives in how to put eval gates on a Claude agent. This note is the CI shape.
What to block on
Block merge when:
- A safety or injection case fails
- A write-path case fails (wrong tool, missing fields, skipped review)
- Parse errors on structured outputs jump
Warn when tone or optional citations move. If everything is blocking, people will disable the suite. If nothing is blocking, you do not have a gate.
Keep the runner boring
- Versioned dataset in the repo
- Deterministic assertions first, judge model second
- Same command locally and in CI
- Artifacts: which cases failed, not a single blended score
Freshness
Add a case the day a customer hits a failure. That is how the suite stays honest. A frozen set from the prototype will not catch the bug you shipped last Thursday.
This is included from week one on production builds and is the core of the eval harness engagement. Instinct is not a release process.
Related notes
Quality and cost
Claude Agent Observability: Trace Model Calls, Tools, Cost, and Failures
A production guide to tracing Claude model calls, tools, cost, failures, and task outcomes with OpenTelemetry-shaped telemetry.
Quality and cost
Put eval gates on your Claude agent before you scale traffic
An eval harness turns Claude quality into a CI number. Here is the minimum setup that catches regressions before customers do.
Quality and cost
Prompt caching is how most Claude bills actually come down
Uncached long context is the usual 3x overspend. Cache the stable prefix, budget the rest, and measure cost per successful task.