A Prompt Edit Broke 9 Test Cases. The Merge Gate Caught It.

A one-paragraph prompt edit passed review and broke 9 test cases. Prompts are code, so I built a gate that treats them that way: every change is re-run against a golden dataset, and a critical regression blocks the merge.

The pull request that must not merge

The project is an email classifier. Its behavior lives in a prompt, a model name and a config file, none of which a compiler will ever complain about.

On PR #5 I changed one paragraph of the prompt. The edit told the model that emails about plans, seats, members and subscriptions belong under billing. It read like a clarification, and it contradicted an existing precedent: those emails were supposed to be account.

Here is what the gate reported:

• Overall pass rate: 93.0% down to 88.0%
• Cases regressed: 9
• Deciding signal: the account category dropped 20.0 points, against a critical threshold of 0.20
• Result: the required eval-gate check went red and the merge was blocked

I reverted to the previous prompt version, the pass rate recovered to 92.0%, and the check turned green. Then I closed the PR unmerged, because it only existed to prove the gate works.

You can look at the red-then-green checks tab on GitHub yourself. Nothing on this page is a mock-up.

Why a prompt edit is the dangerous kind of change

A code change that breaks behavior usually fails loudly. A prompt change fails quietly, and in one category at a time. An average accuracy number can stay almost flat while one slice of your traffic gets worse. In this case the overall drop was 5 points. The account drop was 20.

Building a gate around that problem meant satisfying a few forces at once:

1. It has to see per-category damage, not only the average.
2. LLM output is noisy. Two identical runs disagree with each other, so “any difference blocks” would block everything.
3. The grader is itself a model, and I had to find out whether to trust it.
4. It runs on every relevant PR and costs real money, so a runaway run is a bug.
5. It runs on untrusted input. A CI job that holds an API key is an attack surface.

Every decision below is an answer to one of those forces.

The foundation under all of it is the golden dataset: 100 hand-curated, individually reviewed emails for an invented SaaS product, 25 per category, with 24 of them deliberately hard. It is small, and that has an upside: a dataset I can review case by case is one I can trust when a case flips, and a flipped case is the unit the whole gate reasons about. A bigger, sloppier set would give me a more impressive number and a less believable verdict.

The system at a glance

PR touches prompt / model / config / dataset / classifier code
        |
        v
  build the image in the CI job
        |
        v
  run the 100-case golden set
        |
        v
  diff against the latest main baseline
        |
        v
  verdict --> required check (block / pass)
         --> Slack alert
         --> PR comment
         --> run appended to the eval-history branch

Only PRs that touch a behavior path run the eval. Everything else gets an instant, free pass. The golden set is 100 hand-curated cases, 25 per category, 24 of them deliberately hard.

Run history lives on a dedicated eval-history branch, one JSON file per run. I considered the CI cache (7-day eviction means silent history loss), a committed SQLite file (two concurrent jobs make an unresolvable binary conflict) and a single append-only file (same conflict, different format). One file per run has nothing to conflict over.

A pull request diffs against the newest qualifying run on main, so every merged run becomes the next PR’s baseline. The baseline is always from the same dataset version, because comparing only the cases two dataset versions happen to share would blame the prompt for a change the dataset caused.

I also wanted to be able to say the gate runs the same thing everywhere. Identical LLM outputs are impossible, as the noise numbers below show, so I compare environment fingerprints instead: Python version, package versions and input hashes. My laptop image and the CI-built image match on every field except CPU architecture (aarch64 versus x86_64). That is a claim I can check, which beats “it works on my machine”.

BOSS FIGHT 1:The judge may warn, but it may never block

The threat. A gate that blocks good PRs gets switched off. A gate that trusts a bad grader blocks the wrong ones.

The options. Let the LLM judge drive the critical verdict, or let only a deterministic check do it.

What I chose. Only the deterministic category match can raise a diff to critical. The judge can raise a warning, and nothing more.

The reason is a measurement, not a feeling. I compared the judge against a human reviewer’s labels and it agreed 8 times out of 12, which is 66.7%, with a systematic bias toward concise answers. The SDK I use also dropped the temperature parameter, so I can’t even pin the judge’s noise down. A signal that unreliable should not hold a veto.

What it costs. A regression that only the judge can see never blocks anything. It shows up as a warning that a human has to read. Twelve calibration cases is also a small sample, so this is a decision I expect to revisit once the judge is recalibrated.

The proof. Look at the Slack alert above. The mean judge score moved from 0.739 to 0.722. The deciding signal was per_category.account, the deterministic one.

BOSS FIGHT 2:Thresholds come from measured noise

The threat. Too tight and the gate cries wolf. Too loose and it misses the thing it exists to catch.

The options. Pick a round number that feels right, or measure the noise first. I measured.

I ran the identical configuration twice, same prompt, same model, same dataset. Overall pass rate differed by 0.00, but the runs flipped 3 cases from pass to fail and 3 from fail to pass, and 47 of 89 judge scores changed by up to 0.4. None of that is a regression. All of it is the noise floor a threshold has to clear.

What I chose. Per-category warning at a 0.12 drop and critical at 0.20. Overall pass rate warns at 0.05 and blocks at 0.10. The largest per-category swing between identical runs was 0.04, so the critical line sits five times above the noise. Every value in config.yaml has a comment recording what it was measured against.

Two ideas I rejected on the evidence. Flip counts as a severity signal: identical runs already produce about 3 flips each way, so it would be a noisy trigger. And a diff that can only warn: before merge it is the only signal that exists, so a warning-only diff leaves the gate with nothing to block on.

What it costs. The thresholds come from two identical runs, not a large history. They are marked for retuning as real runs accumulate, and a small dataset (25 cases per category) means a 20-point swing is 5 cases. This catches a regression. It does not support a real-world accuracy claim.

BOSS FIGHT 3:Fail closed, spend a budget, trust no forks

The threat. The eval job holds an Anthropic API key and a Slack webhook, and it runs on pull requests, which means it can run code someone else wrote.

The options. The tempting one is pull_request_target, which would let fork PRs get a real eval run. I rejected it outright. It runs workflow code with the base repo’s secrets against untrusted fork content, which is a textbook way to leak a key.

What I chose. Fork PRs skip the eval entirely, and that is enforced twice: the gate decision skips them, and an independent check refuses to build a run context for one. If the first layer has a bug, the second still holds.

Cost gets the same two layers. A pre-flight estimate refuses an over-budget run before a single API call. A mid-run guard stops it if spend climbs past the ceiling. A full run costs about $0.45 and the hard ceiling is $2.00. An aborted run is never saved and never counted as a pass.

One failure mode surprised me. A workflow-level path filter looks like the obvious way to skip docs-only PRs, but then the required check never reports, and the PR hangs on “Expected, waiting for status” forever. A job-level condition reports as skipped, which GitHub counts as a pass, at zero cost. The cheap-looking option would have deadlocked every docs PR.

What it costs. Contributors from forks never see an eval result on their PR. For a solo project that is a fair trade, and I wrote it down as one.

What I got wrong, and what I’d change

I decided on AWS before I knew what the system did. My first architecture decision was a go on a cloud deployment, made in the first two days with an empty repo. By the time the pipeline existed, the evidence was clear: the gate’s value is entirely pre-merge, drift detection only needs CI history, and a container image that fingerprints identically on my laptop and in CI had already proven portability. Moving the runner to Lambda would have re-hosted a check that runs in the right place, at the right moment, without catching one more regression. So I reversed it and wrote the reversal up as its own record. Infrastructure added to be shown, and not because the system needs it, is a signal I don’t want to send.

The limits are real.

• The judge is not trusted to gate yet. Recalibrating it is the next job.
• The dataset is 100 fictional cases for an invented product. Enough to catch a regression, not enough to claim real-world accuracy.
• I have only proven the “prompt” half of “prompt or model change”. A model-swap proof is still to do.
• Evaluating live production traffic is a different system, and it belongs in its own project.

Takeaways

• Treat a prompt like code: version it, test it, and gate the merge.
• Measure your noise before you set a threshold.
• Do not give a grader you have not calibrated a veto.
• Fail closed, cap the spend, and never hand secrets to a fork.
• Write down the decisions you reverse. They are the interesting ones.

The full code, the ADRs behind each decision and the CI runbook are in the llm-regression-guard repo. If you have run into a regression this kind of gate would have missed, I’d like to hear about it.