The note
Classic software regression testing asserts exact outputs; LLM regression testing asserts stable behavior. The unit is a scenario — a real input with an expected behavior — and the assertion is semantic: did the answer stay grounded, complete, on-policy, within budget? Exact-match tests give false alarms on every rewording; no tests give silence on every real regression. Scenario suites with semantic scoring are the middle that works.
The discipline is unglamorous and it compounds: record baseline scores before launch; re-run the suite on every change, including the ones "too small to matter" (those are the ones that matter); and convert every production failure into a new scenario, so the suite grows with the system. In my conversational AI work this is exactly what raised release confidence and cut production regressions — the case study behind the testing-rigor note.
Costs stay sane because the scoring is layered: deterministic checks first, a calibrated judge model for the semantic middle, humans only on flagged edges. The full setup — golden sets, scorecards, drift checks, cost and latency budgets — is in the LLM evaluation and testing guide.