The note
The pattern teams get wrong is treating the judge as an oracle. Judges have the same failure modes as the systems they score: rubric ambiguity produces inconsistent verdicts, verbose outputs get scored as better, and a judge can be confidently wrong about factuality it has no way to check. Uncalibrated judge scores drift exactly like the model behavior they're supposed to police.
The working discipline is three rules. First, deterministic checks before the judge: anything verifiable in code — formats, forbidden actions, required fields — never goes to a model for an opinion. Second, calibrate the judge against human-labeled examples, and re-calibrate when you change the judge's model or rubric; agreement with humans is the judge's own test suite. Third, route the judge's low-confidence and high-stakes verdicts to human review — the judge is the workhorse, humans are the ground truth.
Placed that way, the judge becomes the layer that makes regression testing affordable and agent evaluation continuous: it scores every release against the golden suite and a sample of live traffic, and flags exactly the cases worth a human's minutes. That's the "judge" layer of the AI Agent Production Stack — one layer, not the whole stack.