Regression-test your RAG service with DeepEval — and settle a model debate with data

++

0/6

🎯 Skill path0/6 earned

AI evals & observability

• 1Prove a model works
• 2Trustworthy JSON
• 3Real test data at scale
• 4Observe & score production
• 5RAG, end to end
• 🏆Catch regressions in CI

Finish this tutorial, then mark it complete below to earn the skill.

Our GPT-5.6 token-economics analysis ended with a challenge: benchmark promises are a hypothesis, not a reason — run the eval on your workload before you believe a "fewer tokens per task" claim. This tutorial is us taking our own advice.

We take the RAG docs assistant from part 5, wrap it in a DeepEval regression suite — containerized, judged by gemma4 on the WEC Inference API, zero OpenAI dependency — and then use that suite to answer a genuinely open question: should we swap the service's generation model? Qwen2.5-3B-Instruct (current) vs gemma4 (candidate), pass rate and tokens per task, measured head to head.

Spoiler: the eval saved us from a pointless migration. And along the way we hit four real production failures — a disk-full death spiral, a startup race condition, a Docker Compose variable trap that silently evaluated the wrong model, and the fix that makes that class of bug impossible. Every command, number, and error below is from a real run.

What we're building

Three design decisions up front:

1. The eval runner is a container, not a venv. The suite you run locally is byte-for-byte the artifact CI runs — docker compose run eval in both places. (Our first attempt was a venv. It died to a full disk before installing anything; see Troubleshooting.)
2. The judge is gemma4 on WEC. DeepEval assumes OpenAI by default; we give it a 30-line custom model class instead. Questions, answers, and verdicts never leave our infrastructure.
3. We reuse part 5's golden dataset unchanged. The promptfoo tests.csv embeds each reference fact inside an llm-rubric string — one regex pulls it back out. Your eval data outlives your eval framework.

Prerequisites

• The running RAG service from part 5 (rag-api answering on :8000, eval/tests.csv present).
• A WEC Inference API key (Inference → API Keys).
• ~2 GB free disk. Seriously — check df -h / now. Ours read 100% and the story of why is in Troubleshooting.

Step 0 — smoke-test the system under test

Never eval a service you haven't poked by hand first:

curl -s -X POST localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "How do I create a compute instance on WEC?"}' | python3 -m json.tool

Figure 1. The system under test, alive: /ask answers from the WEC docs corpus and cites its sources. Never eval a service you haven't poked by hand first.

Step 1 — the containerized eval runner

eval/Dockerfile:

eval/Dockerfile

FROM python:3.12-slim

WORKDIR /eval

RUN pip install --no-cache-dir deepeval==3.*

# Suite code is mounted at runtime so edits don't need a rebuild
CMD ["deepeval", "test", "run", "test_rag_regression.py"]

Add the service to docker-compose.yml inside the services: block (part 5's file ends with a top-level volumes: section — appending blindly puts your service in the wrong block; docker compose config --quiet catches it):

docker-compose.yml (new service, inside services:)

  eval:
    build: ./eval
    profiles: ["eval"]
    env_file: .env
    environment:
      RAG_API_URL: http://rag-api:8000
    volumes:
      - ./eval:/eval
    depends_on:
      - rag-api

profiles: ["eval"] keeps it out of a normal docker compose up — it only runs when you ask. Build it:

docker compose --profile eval build eval

Figure 2. The eval runner builds in ~42s: python:3.12-slim plus DeepEval 3.x. This exact image is what CI runs later — no venv, no drift.

Step 2 — a judge on your own infrastructure

DeepEval's LLM-as-judge metrics want an OpenAI key. We don't have one and don't want one. eval/wec_judge.py:

eval/wec_judge.py

"""Custom DeepEval judge backed by WiLine Inference (OpenAI-compatible)."""
import os

from openai import OpenAI
from deepeval.models.base_model import DeepEvalBaseLLM

WEC_BASE_URL = os.getenv("WEC_BASE_URL", "https://inference.wiline.com/v1")


class WECJudge(DeepEvalBaseLLM):
    def __init__(self, model: str | None = None):
        self.model_name = model or os.getenv("JUDGE_MODEL", "gemma4")
        self.client = OpenAI(
            base_url=WEC_BASE_URL,
            api_key=os.environ["WEC_API_KEY"],
        )

    def load_model(self):
        return self.client

    def generate(self, prompt: str) -> str:
        response = self.client.chat.completions.create(
            model=self.model_name,
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
        )
        return response.choices[0].message.content

    async def a_generate(self, prompt: str) -> str:
        return self.generate(prompt)

    def get_model_name(self):
        return f"WEC/{self.model_name}"

Prove it answers from inside the compose network before building anything on it:

docker compose --profile eval run --rm eval \
  python -c "from wec_judge import WECJudge; j = WECJudge(); print('judge says:', j.generate('Reply with exactly: WEC judge online'))"

Output

judge says: WEC judge online

Figure 3. "WEC judge online" — gemma4 answering through the container, on the same compose network as the service it will judge. No OpenAI key anywhere in the stack.

An honest caveat we carry through the rest of this tutorial: in the model comparison later, gemma4 is both judge and contestant. The mitigation is structural — the judge only compares answer-vs-reference-fact, never "which model wrote this" — and we watch its verdicts on the known-hard cases for favoritism. (It showed none: every case it failed gemma4 on is a case it also fails Qwen on.)

Skill unlocked 🏅

You can point DeepEval's LLM-judged metrics at any OpenAI-compatible endpoint. Your eval verdicts — questions, answers, reasoning — never leave your infrastructure.

Step 3 — expose what you want to measure

Part 5's /ask returned answer and sources — and threw away the token usage that came back with every completion. For token economics we need it, and for the A/B we need the model to be switchable. Three env vars and two response fields in app/main.py (volume-mounted; uvicorn hot-reloads, no rebuild):

app/main.py (excerpt)

# Generation model is configurable so the eval suite can A/B models.
# Defaults preserve the tutorial-5 setup.
GEN_MODEL = os.getenv("GEN_MODEL", "Qwen2.5-3B-Instruct")
GEN_BASE_URL = os.getenv("GEN_BASE_URL", "https://inference.wiline.com/v1")
GEN_API_KEY = os.getenv("GEN_API_KEY", os.environ["WEC_API_KEY"])

generate() now also returns resp.usage, and /ask reports both:

Output (abridged)

{
  "answer": "To view your billing statements ...",
  "sources": ["..."],
  "model": "Qwen2.5-3B-Instruct",
  "usage": {
    "prompt_tokens": 1238,
    "completion_tokens": 122,
    "total_tokens": 1360
  }
}

Figure 4. The new model and usage fields: 1,238 prompt + 122 completion tokens for one billing question. This is the number the token-economics debate is actually about.

That usage block is the number the GPT-5.6 marketing is about — measured on your workload, per request. Pass the variable through compose (GEN_MODEL: ${GEN_MODEL:-Qwen2.5-3B-Instruct} under rag-api's environment:) and remember this line — it comes back to bite us in Troubleshooting.

Step 4 — the regression suite

eval/test_rag_regression.py. It reads part 5's tests.csv, extracts the reference fact from each promptfoo rubric, asks the live service, logs token usage, and asserts a GEval correctness metric judged by WECJudge:

eval/test_rag_regression.py

"""DeepEval regression suite for the WEC docs RAG service.

Reuses the promptfoo golden set (tests.csv) from the previous tutorial:
the reference fact is parsed out of each llm-rubric string.
"""
import csv, json, os, re

import pytest
import requests
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

from wec_judge import WECJudge

RAG_API_URL = os.getenv("RAG_API_URL", "http://rag-api:8000")
USAGE_LOG = os.getenv("USAGE_LOG", "usage_log.jsonl")
MAX_CASES = int(os.getenv("MAX_CASES", "0"))  # 0 = all

FACT_RE = re.compile(r'Reference fact from the official docs: "(.*?)"')


def load_cases():
    with open("tests.csv", newline="") as f:
        for row in csv.DictReader(f):
            m = FACT_RE.search(row["__expected"])
            yield row["question"], (m.group(1) if m else "")


CASES = list(load_cases())
if MAX_CASES:
    CASES = CASES[:MAX_CASES]

judge = WECJudge()

correctness = GEval(
    name="Correctness",
    criteria=(
        "The actual output must correctly address the input question and must not "
        "contradict the reference fact given as expected output. Different wording "
        "is fine. Fail only if the answer is wrong, contradicts the reference fact, "
        "or fails to answer the question."
    ),
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
    ],
    model=judge,
    threshold=0.5,
)


@pytest.mark.parametrize(
    "question,reference", CASES, ids=[q[:50] for q, _ in CASES]
)
def test_rag_regression(question, reference):
    r = requests.post(f"{RAG_API_URL}/ask", json={"question": question}, timeout=120)
    r.raise_for_status()
    data = r.json()

    expect = os.getenv("EXPECT_MODEL")
    assert not expect or data["model"] == expect, (
        f"suite expected {expect} but service is running {data['model']}"
    )

    with open(USAGE_LOG, "a") as f:
        f.write(json.dumps({
            "model": data.get("model"),
            "question": question,
            **data.get("usage", {}),
        }) + "\n")

    assert_test(
        LLMTestCase(
            input=question,
            actual_output=data["answer"],
            expected_output=reference,
        ),
        [correctness],
    )

(That EXPECT_MODEL assertion wasn't in the first version. It exists because of a bug you'll meet in a moment.)

First run, capped at 5 cases to fail fast:

docker compose --profile eval run --rm \
  -e MAX_CASES=5 -e DEEPEVAL_TELEMETRY_OPT_OUT=YES \
  eval

4 of 5 passed in 2:37 — and both the passes and the failure are worth reading. The failure ("What is the starting point for billing?", score 0.3) isn't a service bug: the answer was factually fine, but the golden label is a section title ("How Billing Works") and the judge wanted it named. Remember this case — it flickers between pass and fail all tutorial long, and that flicker becomes a finding. Meanwhile on "total number of events" the judge scored a 0.6 borderline pass with exactly the right reasoning (the service described the feature instead of giving steps). Nuanced verdicts from a self-hosted judge — the thing people assume you need GPT-4-class models for.

Figure 5. First run: 4 of 5 passed in 2:37, every verdict with written reasoning from the gemma4 judge — including the one failure, a strict verdict on a section-title label rather than a service bug.

Note DeepEval's summary line: token cost: None. It can't price a self-hosted judge — which is exactly why the suite writes its own usage_log.jsonl.

Figure 6. The suite’s own accounting: one JSON line per question — model, prompt, completion, total — the raw data every comparison in this tutorial is built from.

Skill unlocked 🏅

You have a regression suite grading a live RAG service — reusing last tutorial's golden set unchanged, judged by a self-hosted model, with per-question token accounting the framework itself doesn't provide.

Step 5 — the experiment: Qwen2.5-3B vs gemma4

The question the suite exists to answer: our service generates with Qwen2.5-3B-Instruct — should it? gemma4 is newer, well-regarded, already in the WEC catalog. Feels like an upgrade. Feels. Let's measure.

Baseline, all 16 cases:

docker compose --profile eval run --rm \
  -e DEEPEVAL_TELEMETRY_OPT_OUT=YES \
  -e USAGE_LOG=usage_qwen.jsonl \
  eval

14/16 (87.5%) in 8:50. The two failures are label artifacts, not RAG bugs: the golden facts from part 5 are section titles ("How Billing Works"), and the judge dinged answers that were factually fine but didn't echo the title; and mp3 vs the label audio/mpeg — the same fact at two abstraction levels, ruled a contradiction. Read your failures before trusting your pass rate. We leave both in: they're honest label debt, and they become our judge-favoritism canary in the A/B.

Figure 7. The baseline: 87.5% in 8:50. Both failures are label artifacts, not RAG bugs — read your failures before trusting your pass rate.

Challenger — same suite, one env var (prefix it on every compose command; Troubleshooting explains why in blood):

GEN_MODEL=gemma4 docker compose up -d rag-api
until curl -sf localhost:8000/health > /dev/null; do echo "waiting..."; sleep 3; done
curl -s -X POST localhost:8000/ask -H "Content-Type: application/json" -d '{"question": "ping"}' \
  | python3 -c "import json,sys; print('>>> model is:', json.load(sys.stdin)['model'])"

GEN_MODEL=gemma4 docker compose --profile eval run --rm \
  -e DEEPEVAL_TELEMETRY_OPT_OUT=YES \
  -e USAGE_LOG=usage_gemma4.jsonl \
  -e EXPECT_MODEL=gemma4 \
  eval

And verify the log, because run labels lie:

sort <(cat eval/usage_gemma4.jsonl | python3 -c "import json,sys; [print(json.loads(l)['model']) for l in sys.stdin]") | uniq -c
#     16 gemma4

Figure 8. Switching the challenger in: GEN_MODEL=gemma4, wait for /health, then confirm >>> model is: gemma4 from the response itself — before spending a single eval minute.

Figure 9. The gemma4 run: 81.25% in ~13 minutes — and its three failures are the same label-artifact cases Qwen flickers on. No favoritism from the gemma4 judge toward the gemma4 contestant.

The verdict table

One script over the usage logs:

run	pass rate	avg prompt tokens	avg completion tokens	total tokens
Qwen2.5-3B (run 1)	87.5%	1,126	75	19,225
Qwen2.5-3B (run 2)	81.25%	1,126	75	19,225
gemma4	81.25%	1,199	415	25,834

Figure 10. The verdict in five lines: same pass rate, 5.5× the completion tokens. The eval just prevented a pointless migration.

Two findings, one expected and one not:

The verdict: don't switch. Same effective quality, same failure cases — but gemma4 uses 5.5× the completion tokens (415 vs 75 per answer) and runs ~45% slower per question. On a RAG workload where the ~1,100-token retrieved context dominates the prompt, the completion column is the whole cost difference. "Newer model" lost to "measure it" in one afternoon. This is precisely the eval the GPT-5.6 news post said to run — and it's the same eval you'd run against GPT-5.6 itself: change GEN_MODEL, GEN_BASE_URL, and GEN_API_KEY, nothing else.

The judge has a noise floor. Look at the two Qwen rows: identical token counts — 19,225 in both runs, because temperature-0 generation is deterministic — yet 87.5% vs 81.25%. The answers didn't change; only the judge's verdicts did (one borderline 0.6 became a 0.3). That cleanly isolates judge nondeterminism from model behavior, and it sets your CI threshold: a gate at "≥ 87.5%" would flake. Ours tolerates one flipped verdict and fails on two or more real regressions.

Figure 11. Qwen run 2: the same 16 deterministic answers, 81.25% this time — the judge, not the model, is where the variance lives.

Skill unlocked 🏅

You can settle a model-swap debate with measured data — pass rate and tokens per task — and you know your judge's noise floor, so you can set a CI threshold that won't flake.

Step 6 — the CI gate

The payoff of containerizing everything: the CI job is the command you've been running all along. .github/workflows/eval.yml:

.github/workflows/eval.yml

name: rag-regression-eval
on:
  pull_request:
    paths: ["app/**", "eval/**"]
  workflow_dispatch:

jobs:
  eval:
    runs-on: [self-hosted, wec]   # runner with access to inference.wiline.com
    steps:
      - uses: actions/checkout@v4
      - name: Run regression suite
        env:
          WEC_API_KEY: ${{ secrets.WEC_API_KEY }}
        run: |
          docker compose up -d rag-api
          docker compose --profile eval run --rm \
            -e DEEPEVAL_TELEMETRY_OPT_OUT=YES \
            -e EXPECT_MODEL=Qwen2.5-3B-Instruct \
            eval
      - name: Teardown
        if: always()
        run: docker compose down

Any PR that touches the app or the eval must survive 16 judged questions against the live service — with the model identity asserted — before it merges.

Skill unlocked 🏅

Your eval is a CI gate. The container you ran locally is byte-for-byte what CI runs — no "works on my machine" gap between development and the pipeline.

Troubleshooting — the four real failures

[Errno 28] No space left on device — and it came back

Our first pip install deepeval died: disk 100% full, 58G of 58G. We freed 2 GB of journal logs and build cache… and it was full again minutes later. The culprit:

Output (the culprit)

7.9G  /var/lib/docker/containers/10281d.../10281d...-json.log

langfuse-clickhouse-1 was in an error loop, spraying stack traces into a Docker json-log with no rotation configured — a death spiral, because a full disk causes more errors causes more log. It had been "unhealthy" for 8 days. The fix, in order (order matters — truncating while the container runs just refills):

docker stop langfuse-clickhouse-1
sudo sh -c "truncate -s 0 /var/lib/docker/containers/10281d*/*-json.log"

Then make it permanent — log rotation in the compose override, applied to every service:

docker-compose.override.yml (Langfuse stack)

x-logging: &default-logging
  logging:
    driver: json-file
    options:
      max-size: "50m"
      max-file: "3"

services:
  clickhouse:
    <<: *default-logging
  # ... every other service

docker compose up -d to recreate. ClickHouse went healthy for the first time in 8 days. Docker's json-file driver does not rotate by default; on a box running LLM infrastructure that logs enthusiastically, unbounded container logs are a time bomb.

13 tests fail with Connection refused, then 3 pass

Changing GEN_MODEL recreates rag-api, and the app takes ~3 minutes to boot (Chroma + embedding model load). Our suite started hammering /ask immediately: 13 straight connection-refused failures, then the API came alive mid-run and the last 3 passed. Flaky in the worst way — the failure depends on wall-clock timing.

Fix it in the suite, not in human discipline — eval/conftest.py:

eval/conftest.py

"""Refuse to start the suite until the RAG API is healthy."""
import os, time

import requests

RAG_API_URL = os.getenv("RAG_API_URL", "http://rag-api:8000")
READY_TIMEOUT = int(os.getenv("READY_TIMEOUT", "300"))


def pytest_configure(config):
    deadline = time.time() + READY_TIMEOUT
    while time.time() < deadline:
        try:
            requests.get(f"{RAG_API_URL}/health", timeout=3).raise_for_status()
            return
        except requests.RequestException:
            time.sleep(3)
    raise RuntimeError(f"rag-api not healthy after {READY_TIMEOUT}s — aborting eval")

The compose variable trap: we evaluated the wrong model. Twice.

The nastiest one, because nothing errored. ${GEN_MODEL:-Qwen…} in compose is interpolated from your shell at every invocation. We started the service with GEN_MODEL=gemma4 docker compose up -d — correct. Then ran the eval without the prefix. Compose resolved the variable back to the Qwen default, saw the running container as config drift, and — via depends_on — silently recreated rag-api with Qwen. The run completed, the summary said "gemma4" in every verdict (the judge was gemma4), and every answer had been written by Qwen.

We only caught it because the suite logs the model from the response:

sort <(... usage_gemma4.jsonl ...) | uniq -c
#     16 Qwen2.5-3B-Instruct     ← the "gemma4" run

Never trust the label on a run; verify from data the system under test reports about itself. The permanent fix is the EXPECT_MODEL assertion in the suite (Step 4). Proof it works — the deliberate-failure test:

Output

FAILED ... AssertionError: suite expected gemma4 but service is running Qwen2.5-3B-Instruct
1 failed in 2.44s

Under three seconds, before spending a single judge call — versus the 10+ minutes each mislabeled run cost us.

Figure 12. The guard in action: wrong model → loud failure in under three seconds, before a single judge call is spent. Compare with the 10+ minutes each silently mislabeled run cost us.

sleep 5 is not a readiness check

Bonus small one: after recreating the container we piped /ask output into a JSON parser 5 seconds later and got json.decoder.JSONDecodeError: Expecting value: line 1 column 1 — the server wasn't up, curl returned empty, the parser choked on nothing. Poll /health in a loop; never sleep a guessed number.

What you built

• A containerized DeepEval regression suite — same image locally and in CI
• A custom judge on WEC Inference (gemma4 + GEval): verdicts with reasoning, no OpenAI key
• Per-question token accounting the framework itself couldn't give you
• A measured model decision: gemma4 ≈ Qwen on quality, 5.5× the completion tokens — keep Qwen
• A judge noise-floor measurement (deterministic answers, a 6.25-point pass-rate swing) that sets your CI threshold honestly
• An EXPECT_MODEL guard that turns silently-evaluated-the-wrong-model into an under-three-second loud failure
• A GitHub Actions gate that runs the identical container on every PR

🎯

Finished this tutorial?

Mark it complete to earn Catch regressions in CI on your skill path.

What's next

The suite judges answers. Part 4's Langfuse traces know why — which chunks were retrieved, what the generation span cost, where latency hides. Wiring eval failures back to their traces (DeepEval test ID ↔ Langfuse trace ID) turns "test failed" into "here's the retrieval that caused it." That's the component-level debugging story, and it's where this series goes next.

Further reading

• DeepEval docs — custom models
• DeepEval — GEval metric
• Docker logging drivers — json-file rotation
• Compose variable interpolation