10 agent evals

after Hanako

10 agent evals for AI engineers explained with usage

1. Golden set

20 cases you never change
18 / 20 passed

2. LLM as judge

a model grades against a rubric
output
draft
judge

3. Rubric scoring

one number per dimension
correctness
tone
safety
cost

4. Trajectory eval

grade the path, not the answer
step 4 called the wrong tool

5. Tool unit tests

test the hands, not the brain
read()pass
write()pass
search()pass
exec()fail
fetch()pass
diff()pass

6. Regression suite

did the new prompt break turn 4
case 3 regressed

7. A/B in prod

real traffic, split live
traffic
A62%B38%

8. Human review

sample it, do not read it all
reviewed

9. Shadow run

the candidate runs, nobody sees it
live
shadow
same input, unseen output

10. Red team

try to break it before they do
jailbreakexfilprompt injecttool abuse
1 got through
agent