Gating Prompt Changes in CI

A one-line prompt tweak is the most dangerous kind of change an LLM system sees, precisely because it doesn't look dangerous. There's no type error, no failing import, no obviously broken syntax — just a sentence that reads slightly better to the person editing it. Whether that sentence quietly changes the model's tool-selection behaviour, or makes it 20% more likely to answer from memory instead of calling a tool, is something you generally find out from user complaints, days later, in production. That gap — between "the diff looks fine" and "the behaviour changed" — is what an offline evaluation harness is for.

Prompts are code; treat changes like code changes

The premise here isn't exotic: any other change to application logic goes through a test suite before merge. Prompts control agent behaviour just as much as application logic does, so they should clear the same bar — a required check, gating the merge, not a suggestion a reviewer might remember to run manually.

The harness has two layers, doing different jobs:

Regression tests are deterministic fixtures — a fixed input, a fixed expected property of the output (a specific tool got called, a specific field is present, a specific value falls in range). These catch hard regressions: a prompt edit that breaks tool-selection on a case that used to work, a schema change that drops a field the downstream code expects. Cheap to run, unambiguous to interpret, and they should never go red on a change that didn't intend to affect that behaviour.

LLM-as-judge scoring covers everything regression tests structurally can't: is this answer actually good, on axes that don't reduce to an exact-match assertion? A separate judge model scores each response against a rubric — factual grounding, completeness, tone, whether the response over-claims relative to what the retrieved evidence actually supports.

JUDGE_PROMPT = """
You are evaluating an AI assistant's response for a clinical query system.
 
Query: {query}
Retrieved context: {context}
Response: {response}
 
Score 1-5 on each axis:
- Grounded: Is every claim supported by the retrieved context?
- Complete: Does it address the full query?
- Hallucination: Does it state anything not in the context? (5 = none, 1 = severe)
 
Return JSON: {{"grounded": int, "complete": int, "hallucination": int, "reasoning": str}}
"""

Two design choices matter more than the prompt wording itself. First, the judge is a different, typically stronger model than the one being evaluated — using the same model to grade its own output tends to inherit that model's specific blind spots instead of catching them. Second, the judge always sees the retrieved context alongside the response, not the response in isolation, because "hallucination" is only a meaningful score relative to what evidence the system actually had available — a claim can be correct and still be a hallucination if nothing in the retrieved context supported it.

The three metrics that actually get tracked

Distilled down, the harness tracks three things across every run:

• Answer quality — the LLM-as-judge rubric scores, aggregated across the fixture set, watched for regression against the current baseline rather than an absolute pass/fail threshold.
• Hallucination rate — the fraction of responses making claims unsupported by retrieved context. This is the metric a bad prompt change is most likely to move, and the one manual review is worst at catching, since a hallucinated clause often reads as perfectly fluent.
• Tool-selection accuracy — for agentic flows, whether the system called the tool a labelled fixture says it should have called. This is what catches an intent-router prompt edit that quietly biases classification toward the wrong branch.

Gating it in CI, not just running it

A harness that exists but isn't wired into the merge path is a harness someone remembers to run when they think to. The eval suite runs as a required GitHub Actions check on any change touching prompts or model configuration:

name: LLM Eval Gate
on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/agents/**"
 
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - name: Run regression fixtures
        run: pytest tests/eval/regression -v
      - name: Run LLM-as-judge suite
        run: pytest tests/eval/judge --threshold-file=baselines/current.json

The --threshold-file step is what turns "run some evals" into an actual gate: judge scores get compared against a checked-in baseline, and the job fails if quality drops beyond a set tolerance, or if hallucination rate climbs. A prompt change that improves the wording but tanks grounding on three fixtures fails the build the same way a broken unit test would — before a reviewer ever has to notice the regression by reading the diff carefully enough to feel it.

Where this breaks down, and why it's still worth doing

The obvious weakness: LLM-as-judge is itself a model making judgment calls, so the harness inherits some of the failure modes it's meant to catch — a systematically biased judge produces a systematically biased gate. In practice this is managed rather than solved: periodic manual spot-checks against the judge's scores, rubrics narrow and concrete enough that the judge has less room to drift, and treating judge scores as a regression signal (did this get worse relative to baseline) rather than an absolute quality certification.

That's a real limitation, not a footnote. But the alternative isn't a perfect evaluation system — it's no gate at all, and a prompt regression that ships silently into a clinical chatbot. An imperfect gate that catches most regressions before merge is a large improvement over a review process that catches none, and it's the difference between "we found out from CI" and "we found out from a user."