How to Write AI Evals That Catch Real Bugs (7 Rules That Work)

Skip vague 1–5 scoring rubrics. Use these seven practical rules to build fast, deterministic evals that catch regressions before you ship a prompt or model change.

Abstract editorial still life of a measuring probe inspecting a cracked module in a row of identical blocks

Most teams delay writing evals because they picture a massive academic benchmark. In practice, a 30-case regression suite built from your own bug reports will catch 90% of production regressions before users ever see them. Here are seven rules for writing evals that actually pay off.

1. Harvest Cases from Real Failure Logs

Don't invent toy inputs in a vacuum. Every time an agent picks the wrong tool, drops a citation, or hallucinates a parameter in staging or production, turn that exact trace into a permanent eval case.

2. Prefer Binary Pass/Fail Over 1–5 Scales

A 1–5 "helpfulness" score hides regressions inside averages. Write crisp, binary assertions: Did it cite a valid path? Did it call the allowlisted tool? Did it refuse when the record was missing? Binary checks make pass rates immediately actionable.

3. Use Deterministic Assertions First, LLM Judges Second

Run cheap, deterministic checks before calling an LLM judge: JSON schema validity, regex/exact-match checks on IDs, tool-call counts, and verifying that every cited URL or filename actually appeared in the tool outputs. Reserve LLM-as-judge strictly for tone, completeness, or semantic equivalence.

4. Grade the Trajectory, Not Just the Final Answer

An agent that guesses the right answer after calling a forbidden tool or hitting five unnecessary retries is still broken. Assert on the sequence of tool calls, argument bounds, and iteration counts.

5. Keep a 25% Refusal and Edge-Case Slice

Models are trained to be agreeable, which means they love fabricating answers when context is missing. Dedicate at least a quarter of your suite to unanswerable queries, conflicting records, and malformed inputs where the only passing behavior is an honest refusal or clarifying question.

6. Keep Your Core Suite Fast Enough to Run on Every Change

If an eval suite takes 20 minutes and costs $15 per run, nobody will run it. Maintain a tight 25–40 case smoke suite that finishes in under a minute so you can run it before every prompt edit, tool schema tweak, or model upgrade.

7. Never Leak Eval Cases into System Prompts

If you fix a failing eval by pasting that exact example into your system prompt's few-shot block, you're overfitting to the test. Generalize the rule in your prompt, and add two fresh variations of the failure to your eval set.

More stories