LLM Regression Guard
A required CI check for LLM features. A one-paragraph prompt edit dropped one category by 20 points; the gate caught it and blocked the merge.
#01_GUARD
Complete / [Merge Gate Pipeline]
LLM Regression Guard treats a prompt like code. Whenever the prompt, model, config or dataset of an email classifier changes, a CI job re-runs a fixed 100-case golden dataset, compares the result with the latest main-branch baseline, and reports through a required check, a Slack alert and a PR comment.
It was built to show what separates a demo from a product: measured thresholds, a grader that is not trusted blindly, a hard cost ceiling, and a pipeline that holds up on untrusted pull requests.
Sys_specs
| Trigger | Prompt, model or config PR |
|---|---|
| Golden set | 100 cases |
| Gate | Required check |
| Cost per run | ~$0.45 |
| History | Append-only branch |
Diffs every run against the latest main baseline and blocks the merge when a category drops past a threshold.
Set from two identical runs, so the gate does not cry wolf on normal LLM variance.
The LLM judge agreed with a human only 66.7% of the time, so only a deterministic check can gate.
A $2.00 ceiling is enforced before and during a run, and fork PRs never see the API key.
✳ Threat: Silent Regression
A prompt edit that reads like a harmless clarification can quietly degrade one category while the overall average barely moves.
Remedy: Gate on the Diff
Re-run a hand-curated golden dataset on every behavior change, diff per category against main, and block on a critical drop.