4.1Beginner30 minutes
4.1 — Define a balanced metric pack for a support copilot
What you will learn
- Name quality, latency, cost, safety, and security metrics
- Set floors/ceilings with owners
- Tie metrics to user outcomes
← Build Exercises
Domain 4
Evaluation, Testing & Optimization
6 build exercises to practice the concepts in this domain.
4.2Intermediate40 minutes
4.2 — Design a mixed-method eval suite for policy RAG
What you will learn
- Cover happy, edge, and adversarial cases
- Mix automated and human labels
- Version the dataset with the system
4.3Intermediate35 minutes
4.3 — Plan a controlled A/B for a prompt change
What you will learn
- Isolate one primary factor
- Define power/traffic assumptions
- Write ship criteria for quality and cost
4.4Advanced40 minutes
4.4 — Diagnose a failing golden set with a decision tree
What you will learn
- Reproduce with fixed inputs and traces
- Separate prompt, retrieval, grounding, and model mismatch
- Pick the smallest justified fix
4.5Intermediate40 minutes
4.5 — Optimize a hot-path assistant under a quality floor
What you will learn
- Profile tokens and latency
- Apply trim and prefix cache before tier changes
- Re-measure quality after each cut
4.6Advanced40 minutes
4.6 — Design quality monitoring with an alert→fixture loop
What you will learn
- Define quality/cost SLIs beyond uptime
- Plan redaction and request_id joins
- Wire alerts to runbooks and eval capture