The capstone: build, evaluate, and observe a RAG docs assistant on the WEC API
+
Everything in this series was building to this. You can prove a model works (part 1), make its output machine-reliable (part 2), generate real test data (part 3), and observe production (part 4). Now we spend all four skills at once on the pattern behind almost every serious LLM product: RAG — retrieval-augmented generation.
We'll build a docs assistant: a containerized HTTP service that answers customer questions from WEC's own documentation. Not a notebook — a service, Docker-first, the shape you'd actually deploy. And because this series doesn't do happy-path demos: along the way our RAG hallucinates a GPU price, we root-cause it to our own scraper, fix it, and pin the fix with a regression test. Every command, number, and error below is from a real run.
What we're building
One architecture decision up front: we split the work. Embeddings run locally (a small ONNX
model — fast, free, no GPU needed) and generation runs on WEC. This is a completely standard
production split: retrieval is cheap and latency-sensitive, so keeping it next to your vector
store saves a network round-trip per query — while generation is where the big model earns its
keep. (When an embedding model lands in the WEC catalog, moving that piece over is a two-function
swap — point the same OpenAI client at /v1/embeddings in ingest.py and main.py; everything
else in this tutorial stays as is.)
Prerequisites
- Docker + Compose on a WEC Instance (the same setup as part 4's Langfuse box works).
- A WEC Inference API key (Inference → API Keys).
- Your part-4 Langfuse instance and keys (optional but recommended — used in Step 3's tracing).
- ~1 GB free disk for the image. Check first:
df -h /. Our box was at 99% and survived, but tight disk is the #1 silent killer of Docker builds.
Step 1 — Scaffold the service, Docker-first
Real RAG features ship as services, so we start as one: a FastAPI app in a container, code
bind-mounted so uvicorn --reload picks up every edit — no rebuild per change.
mkdir -p ~/rag-service/app && cd ~/rag-service
cat > requirements.txt <<'EOF'
fastapi==0.115.6
uvicorn==0.34.0
chromadb==0.5.23
fastembed==0.4.2
openai==1.59.7
langfuse
requests==2.32.3
beautifulsoup4==4.12.3
EOF
cat > Dockerfile <<'EOF'
FROM python:3.12-slim
WORKDIR /srv
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ app/
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
EOF
cat > docker-compose.yml <<'EOF'
services:
rag-api:
build: .
ports:
- "8000:8000"
volumes:
- ./app:/srv/app # live code - edit on host, uvicorn reloads
- chroma-data:/data # persisted vector index
- model-cache:/root/.cache # embedding model cache (survives rebuilds)
environment:
ANONYMIZED_TELEMETRY: "False"
WEC_API_KEY: ${WEC_API_KEY}
LANGFUSE_PUBLIC_KEY: ${LANGFUSE_PUBLIC_KEY}
LANGFUSE_SECRET_KEY: ${LANGFUSE_SECRET_KEY}
LANGFUSE_HOST: ${LANGFUSE_HOST}
volumes:
chroma-data:
model-cache:
EOF
Secrets live in an .env file next to the compose (compose loads it automatically — keys never
enter the YAML). Keep it out of version control from the start:
cat > .env <<'EOF'
WEC_API_KEY=sk-your-wec-key
LANGFUSE_PUBLIC_KEY=pk-lf-…
LANGFUSE_SECRET_KEY=sk-lf-…
LANGFUSE_HOST=http://<your-langfuse-host>:3000
EOF
echo ".env" >> .gitignore
A minimal app so the container has something to serve:
cat > app/main.py <<'EOF'
from fastapi import FastAPI
app = FastAPI(title="WEC docs RAG")
@app.get("/health")
def health():
return {"status": "ok"}
EOF
docker compose up -d --build
curl -s localhost:8000/health
{"status":"ok"}
Figure 1. One command each: the container up, /health answering, and the disk check
(ours reads 99% — this box lives dangerously). The build takes ~90s and a ~700 MB image.
Step 2 — Ingest: build the corpus from a live website
In the real world, the knowledge base you'll want your assistant to answer from usually isn't a tidy folder of markdown — it's a live website: product docs, a wiki, a help center. So the corpus-acquisition skill worth learning is real scraping, not file loading.
For this tutorial we scrape WEC's own documentation site. It's built with Docusaurus, and
Docusaurus (like most site generators) publishes a sitemap.xml — which means you can discover
every page programmatically instead of guessing URLs. The same approach works on any site that
ships a sitemap:
curl -s https://wec.wiline.com/docs/sitemap.xml | grep -c "<loc>" # 109 URLs on the whole site
We filter to the product docs (/docs/cloud_portal/), pull each page, and keep only the
<article> element — Docusaurus wraps the actual content in it, so nav/sidebar/footer never reach
the index:
cat > app/ingest.py <<'EOF'
"""Scrape the WEC docs site -> chunk -> embed locally -> Chroma."""
import re, requests, chromadb
from bs4 import BeautifulSoup
from fastembed import TextEmbedding
SITEMAP = "https://wec.wiline.com/docs/sitemap.xml"
FILTER = "/docs/cloud_portal/" # product docs only
CHUNK, OVERLAP = 700, 100 # chars
def discover():
xml = requests.get(SITEMAP, timeout=30).text
urls = re.findall(r"<loc>([^<]+)</loc>", xml)
return [u for u in urls if FILTER in u]
def scrape(url):
html = requests.get(url, timeout=30).text
art = BeautifulSoup(html, "html.parser").find("article") # Docusaurus main content
return art.get_text("\n", strip=True) if art else ""
def chunk(text):
out, i = [], 0
while i < len(text):
out.append(text[i:i + CHUNK])
i += CHUNK - OVERLAP
return out
if __name__ == "__main__":
urls = discover()
print(f"sitemap -> {len(urls)} product-doc pages")
docs, metas = [], []
for u in urls:
for c in chunk(scrape(u)):
docs.append(c)
metas.append({"url": u})
print(f"scraped -> {len(docs)} chunks")
embedder = TextEmbedding("BAAI/bge-small-en-v1.5") # local, ~66MB ONNX, no GPU needed
vectors = [v.tolist() for v in embedder.embed(docs)]
print(f"embedded -> {len(vectors)} vectors (dim {len(vectors[0])})")
db = chromadb.PersistentClient(path="/data")
col = db.get_or_create_collection("wec-docs")
col.add(ids=[str(i) for i in range(len(docs))],
documents=docs, metadatas=metas, embeddings=vectors)
print(f"indexed -> collection 'wec-docs' now has {col.count()} chunks")
EOF
docker compose exec rag-api python -m app.ingest
sitemap -> 37 product-doc pages
scraped -> 281 chunks
model_optimized.onnx: 100%|████████████| 66.5M/66.5M [00:03<00:00, 16.9MB/s]
embedded -> 281 vectors (dim 384)
indexed -> collection 'wec-docs' now has 281 chunks
Figure 2. The whole product-doc corpus indexed in under a minute. The embedding model caches in
the model-cache volume, so it downloads exactly once.
You can turn any documentation website into a queryable vector index — sitemap discovery, content extraction, chunking, local embeddings.
get_text("\n", …)Our first version used get_text(" ") — a single space as the separator. It looked harmless and
caused a genuine hallucination you'll see in Step 4. Keep the newline; we'll come back to this.
(Want to watch the bug happen on your own box? Change "\n" back to " " in scrape(), re-run
the ingest, and ask the Step-4 question — then put the newline back and re-ingest.)
You may see Failed to send telemetry event ClientStartEvent: capture() takes 1 positional argument… — a known chromadb/posthog version clash. It's harmless (your data is fine), and the
ANONYMIZED_TELEMETRY: "False" env in the compose silences it.
Step 3 — The /ask endpoint: retrieve, generate, trace
Now the service itself. Three design choices worth stating:
- Citations come from retrieval metadata, not the model. The model can hallucinate URLs; the
vector store cannot.
sourcesis deterministic. - The WEC call goes through
langfuse.openai(part 4's drop-in), and each stage gets an@observespan — so every request produces aask → retrieve → generatetree in Langfuse. timeout=60on the client. Part 4's lesson: no default timeout means a stalled backend hangs your service silently.
cat > app/main.py <<'EOF'
import os, chromadb
from fastapi import FastAPI
from pydantic import BaseModel
from fastembed import TextEmbedding
from langfuse import observe, get_client
from langfuse.openai import openai # auto-traces the WEC call
app = FastAPI(title="WEC docs RAG")
db = chromadb.PersistentClient(path="/data")
col = db.get_or_create_collection("wec-docs")
embedder = TextEmbedding("BAAI/bge-small-en-v1.5")
wec = openai.OpenAI(
base_url="https://inference.wiline.com/v1",
api_key=os.environ["WEC_API_KEY"],
timeout=60,
)
class Ask(BaseModel):
question: str
@observe() # span: retrieve
def retrieve(question: str, k: int = 4):
qv = list(embedder.embed([question]))[0].tolist()
res = col.query(query_embeddings=[qv], n_results=k)
chunks = res["documents"][0]
sources = sorted({m["url"] for m in res["metadatas"][0]})
return chunks, sources
@observe() # span: generate (WEC call nested inside)
def generate(question: str, chunks: list[str]) -> str:
context = "\n---\n".join(chunks)
resp = wec.chat.completions.create(
model="Qwen2.5-3B-Instruct", # small on purpose: cheap + fast per query,
# and its failures teach (see Step 5)
name="rag-generate",
messages=[{"role": "user", "content":
"Answer the question using ONLY the context below. "
"If the context doesn't contain the answer, say so. "
"When quoting a price or number, copy it VERBATIM with its unit and the item "
"it belongs to; never combine numbers from different lines and never compute "
"new numbers from examples.\n\n"
f"Context:\n{context}\n\nQuestion: {question}"}],
)
return resp.choices[0].message.content
@app.post("/ask")
@observe() # root span: the whole request
def ask(body: Ask):
chunks, sources = retrieve(body.question)
answer = generate(body.question, chunks)
get_client().flush()
return {"answer": answer, "sources": sources}
@app.get("/health")
def health():
return {"status": "ok"}
EOF
docker compose up -d --build
Ask it a real customer question:
curl -s localhost:8000/ask -H "Content-Type: application/json" \
-d '{"question": "How do I create an API key for the Inference API?"}' | python3 -m json.tool
{
"answer": "To create an API key for the Inference API, follow these steps:\n1. Log in to the WiLine Edge Cloud.\n2. Expand the \"Inference\" section in the left sidebar.\n3. Click on the \"API Keys\" tab.\n4. Click the \"+ Create Key\" button located above the keys table. ...",
"sources": [
"https://wec.wiline.com/docs/cloud_portal/platform/inference/api/wiline-edge-cloud-inference-api/",
"https://wec.wiline.com/docs/cloud_portal/platform/inference/api_keys/",
"https://wec.wiline.com/docs/cloud_portal/platform/inference/examples/",
"https://wec.wiline.com/docs/cloud_portal/platform/inference/models_hub/",
"https://wec.wiline.com/docs/cloud_portal/platform/inference/overview/"
]
}
Figure 3. First try: the exact API-key flow from the docs, with the right page cited. Scrape →
embed → retrieve → generate, end to end.
Open Langfuse → Tracing and you'll find the request as a span tree — ask → retrieve → generate → rag-generate — with per-step latency and tokens. When an answer is bad, this is how you
tell bad retrieval from bad generation (part 4's skill, now on a real app).
Figure 4. Every /ask becomes a tree: retrieval and generation each have an address.
You have a traced, containerized RAG service answering from live-scraped documentation.
Step 4 — The hallucination we caught (for real)
A good RAG must refuse what the docs don't cover. Probe it:
curl -s localhost:8000/ask -H "Content-Type: application/json" \
-d '{"question": "Does WEC offer a free GPU tier for students?"}' | python3 -m json.tool
Our first version (the get_text(" ") scraper) answered:
"No, WEC does not offer a free GPU tier specifically for students. The pricing for
GPU hourly is listed as $0.66/GPU-hr with a 25% off annual prepay option at
$0.10/GB-mo for block storage. There is no mention of a free GPU tier in the
provided information."
The refusal is right. But read the pricing sentence again: "$0.66/GPU-hr with a 25% off annual prepay option at $0.10/GB-mo for block storage" — that's three different facts welded into one false statement, gluing a storage rate onto the GPU prepay option. Check the actual pricing page and you find they're simply three adjacent items:
- GPU hourly: $0.66/GPU-hr
- Annual prepay: 25% off
- Block storage: $0.10/GB-mo
The model stitched a storage price onto "GPU". Why? Look at the chunk we fed it. Our space-separated scrape had flattened the pricing cards into one undifferentiated line:
... $1.44 /mo Starting at $0.66 /GPU-hr GPU hourly 25 % off Annual prepay $0.10 /GB-mo Block storage ...
The hallucination was manufactured by our own ingestion. The model conflated prices because we destroyed the structure before embedding it. This is the most classic RAG failure there is — garbage in, garbage out — and you only catch it by probing with questions and reading the retrieved chunks.
Figure 5. The bug in the wild: correct refusal, garbled pricing — a storage rate glued onto the
GPU prepay option. The sources look perfectly plausible. This is why "it cites sources" is not the
same as "it's grounded."
Two fixes, both already in the code above:
- Structure-aware extraction —
get_text("\n", strip=True)keeps line boundaries, so price cards stay separate lines instead of one soup. - A hardened generation prompt — quote numbers verbatim with unit and item; never combine lines; never derive new numbers from examples. (Before this rule, the model happily computed a fake "$1.5834/GPU-hour" from a daily-spend example.)
After re-ingesting with the fix, the storage-price conflation is gone. Residual ambiguity remains — the pricing strip still contains fragments like "$1.44/mo" (a compute-instance price) near the GPU line, and truly fixing that needs table-aware extraction (see What's next). Which is exactly why the next step exists: pin the fixed bug so it can never quietly return.
Step 5 — Eval the service like a CI gate
This step took four rounds of fixes to get an honest number — and the journey from 39% to 87% is the best lesson in the series, because almost nothing we fixed along the way was the RAG itself.
We reuse part 3's skill — generate a QA dataset from the corpus itself — and grade the live
HTTP endpoint with Promptfoo. The first generator version asked for one question per page with a
key_fact the answer must contain, and got 37/37 pairs back. Some labels already smelled — a vague
"All paid", menu paths instead of answers — but let's run it naively first and let the failures
teach us. Convert to a Promptfoo dataset with plain substring assertions (icontains: + the key
fact) and point it at the running service. The provider is HTTP — we grade the real API,
exactly what a CI gate would do:
mkdir -p eval
{ echo 'question,__expected'; jq -r '[.question, ("icontains:" + .key_fact)] | @csv' app/qa.jsonl; } > eval/tests.csv
cat > eval/promptfooconfig.yaml <<'EOF'
description: "RAG service eval — QA set generated from the scraped docs + regression"
providers:
- id: https
config:
url: http://localhost:8000/ask
method: POST
headers:
Content-Type: application/json
body:
question: "{{question}}"
transformResponse: json.answer
tests:
- file://tests.csv
# regression: the storage-price-as-GPU-price conflation from Step 4
- vars:
question: "What is the pricing model for GPU hourly usage in Edge Cloud?"
assert:
- type: not-icontains
value: "0.10"
- type: not-icontains
value: "GB-mo"
EOF
cd eval && npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
Results:
✓ 15 passed (39.47%)
✗ 23 failed (60.53%)
Duration: 40s (concurrency: 4)
That 39.5% is what we measured during development, while the Step-4 bug was still live. Re-run the same naive eval against the finished service and it lands much higher — 27/38:
Figure 6. The same naive eval against the finished service: 71.05%. Better service, same
noisy labels — the 30-point gap between these two runs is all service quality, and both numbers
mean little until you read the rows.
Read the failures before you panic. From the development run's 23 failures, most are not RAG failures:
| Failure | What actually happened |
|---|---|
| "Balance due" answer: "the amount that remains due… $0.00" | Correct answer; the label demanded the phrase All paid — bad label |
| Payment Information: gives the exact right menu path | Correct; label phrasing doesn't substring-match — brittle assertion |
| GPU pricing: "$0.10 per GB-month" | Real failure — the Step-4 conflation, reproducing systematically |
| TTS default format: "not specified in the context" | Real retrieval miss — the chunk with the default wasn't retrieved |
Upgrade 1 — a judge instead of substrings. It barely helps.
Substring matching (icontains) was perfect for part 2's single JSON field and is too brittle
for long-form RAG answers. The obvious upgrade is part 4's skill: an LLM judge grading
semantic consistency (llm-rubric: assertions + a judge model in defaultTest). We did exactly
that — same 37 labels, judge instead of substrings — and got:
Results:
✓ 17 passed (44.74%)
✗ 21 failed (55.26%)
Duration: 8m 21s (concurrency: 4)
39.5% → 44.7%. Barely moved — because the judge faithfully enforces bad labels. Rows with
garbage key_facts still fail: the reference is wrong, not the answer. A better grader cannot
rescue a bad dataset. The bottleneck was never the assertion type.
Upgrade 2 — curate the labels with a gate
So fix the data (part 3's discipline, now enforced by code). Generator v2 demands the label be
the answer — not a menu path, not a section name — and adds a curation gate: the key_fact
must exist verbatim in the source text, or the row is rejected on the spot:
cat > app/gen_qa.py <<'EOF'
"""Generate a QA eval set from the indexed docs — v2, curated labels."""
import os, json, chromadb
from openai import OpenAI
wec = OpenAI(base_url="https://inference.wiline.com/v1",
api_key=os.environ["WEC_API_KEY"], timeout=60)
db = chromadb.PersistentClient(path="/data")
col = db.get_or_create_collection("wec-docs")
data = col.get(include=["documents", "metadatas"])
pages = {}
for doc, meta in zip(data["documents"], data["metadatas"]):
pages.setdefault(meta["url"], []).append(doc)
kept, skipped = 0, 0
with open("/srv/app/qa.jsonl", "w") as f:
for url, chunks in sorted(pages.items()):
ctx = "\n".join(chunks[:2])[:2000]
resp = wec.chat.completions.create(
model="Qwen2.5-3B-Instruct", temperature=0.3,
messages=[{"role": "user", "content":
"From this documentation excerpt, write ONE question a customer would ask, "
"and the fact that ANSWERS it. Return ONLY JSON with keys:\n"
"question (string), key_fact (string).\n"
"Rules for key_fact: it must be THE ANSWER to the question (not a menu path, "
"not a section name), a short phrase copied VERBATIM from the excerpt, "
"max 8 words.\n\n"
f"Excerpt:\n{ctx}"}],
)
raw = resp.choices[0].message.content
try:
start, end = raw.index("{"), raw.rindex("}") + 1
row = json.loads(raw[start:end])
# curation gate: the label must literally exist in the source text
if row["key_fact"].lower().strip() not in ctx.lower():
print(f"SKIP {url.split('/docs/')[1]}: key_fact not verbatim in source")
skipped += 1
continue
row["url"] = url
f.write(json.dumps(row) + "\n")
kept += 1
print(f"ok {url.split('/docs/')[1]}: {row['question'][:50]} -> {row['key_fact'][:40]}")
except Exception as e:
print(f"SKIP {url}: {e}")
skipped += 1
print(f"done -> {kept} kept, {skipped} rejected by curation gate")
EOF
docker compose exec rag-api python -m app.gen_qa
ok cloud_portal/management/billing/how_billing_works/: What is the starting point ... -> How Billing Works
SKIP cloud_portal/management/billing/overview/: key_fact not verbatim in source
ok cloud_portal/platform/inference/api/wiline-edge-cloud-inference-api/: What is the base URL ... -> https://inference.wiline.com/v1
SKIP cloud_portal/platform/inference/api_keys/: key_fact not verbatim in source
...
done -> 15 kept, 22 rejected by curation gate
15 kept, 22 rejected. That rejection rate is the point: more than half of what the generator produced would have graded the RAG against wrong references. A smaller, trustworthy dataset beats a bigger, poisoned one.
The generator samples at temperature 0.3, so the exact tally shifts slightly between runs — here's a later regeneration that kept 16 of 37; the rejection rate stays brutal either way:
Figure 7. The gate at work: 16 kept, 21 rejected on this run. Every surviving label is provably
in the docs — real answer-facts like audio/mpeg and the API base URL, not menu paths. (The
documented eval runs below use our first curated set: 15 pairs + the regression = 16 tests.)
Rebuild tests.csv with a rubric that gives the judge the question and the verified
reference, plus explicit PASS/FAIL rules (different wording is fine; contradiction fails):
{ echo 'question,__expected'; jq -r '[.question, ("llm-rubric:Question asked: \"" + .question + "\". Reference fact from the official docs: \"" + .key_fact + "\". PASS if the answer correctly addresses the question and does not contradict the reference fact — different wording is fine. FAIL only if the answer is wrong, contradicts the reference, or fails to answer the question.")] | @csv' ../app/qa.jsonl; } > tests.csv
Results:
✓ 12 passed (75.00%)
✗ 4 failed (25.00%)
Duration: 2m 47s (concurrency: 4)
75%. Better — but before celebrating, read the four failures. And here the series' habit pays off one more time, because three of them weren't RAG failures either.
Upgrade 3 — the judge itself was breaking
The three failures whose grader reason was just "No output" all shared two tells:
graderError: true, and a completion of exactly 1024 tokens — Promptfoo's default
max_tokens. Our judge is a thinking model: it spent the whole budget reasoning and got cut off
before emitting the verdict JSON, which Promptfoo counts as FAIL. The fourth "failure" got a
verdict — but the small judge had literally echoed the schema placeholder (reason: "string")
instead of grading. Both answers were actually correct.
Two config lines fix it — a bigger judge, and room to think:
defaultTest:
options:
provider:
id: openai:chat:Qwen3.5-122B
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
# thinking models spend tokens on reasoning before the verdict JSON;
# the 1024 default silently truncated the judge and failed 3 tests
max_tokens: 4096
Results:
✓ 14 passed (87.50%)
✗ 2 failed (12.50%)
And the row that matters most:
│ What is the pricing model for GPU hourly usage in Edge Cloud? │ [PASS] ... starting at $0.66/GPU-hour. │
The regression is green — the Step-4 conflation is pinned and cannot quietly return. (One
cosmetic note: the 122B judge sometimes keeps its reason text to a terse ... — the verdicts
themselves are correct, which is what the gate reads.)
You can gate a live RAG service in CI: generated + gate-curated QA, a judge for semantics, deterministic regressions for known bugs — and you know how to debug the judge itself.
Upgrade 4 — the 87.5% didn't survive a re-run
One habit this series drills: re-run before you believe. We did — and got 75%, with
different rows failing than before. Billing, which had just passed, now failed with a circular
echo; the TTS answer said "mp3" one run and "unspecified" the next. The score was a lottery
because the service itself was non-deterministic: generate() passed no temperature, so
every eval graded a different roll of the dice at the default 1.0.
One line ends the lottery — add to the create() call in generate():
temperature=0, # determinism: same question -> same answer
Re-run twice. 13/16 — 81.25% — both times, with the same three failures. The number went down and that's the win: 87.5% was partly luck (the flaky rows happened to land right that run); 81.25% is reproducible. A reader who re-runs your eval should get your number.
The three real failures — finally, actual RAG bugs
After four rounds of fixing the measurement, what's left is signal — stable across runs, and each one is a different classic RAG failure type:
- Subnets ("Total Networks") — retrieval miss, chunk boundary. The right page ranks, but dump
the top-4 retrieved chunks and none contains the stats block with the answer; the top chunk
starts mid-word (
"iated with subnets\nData Transfer…"). Fixed-size character chunking split the fact across a boundary, and the half with the answer doesn't rank. Fix class: larger overlap, biggerk, or structure-aware chunking. - TTS default format — retrieval miss, fact not in top-k. The chunk stating the default never reaches the context. Revealing detail: at temperature 1.0 this row sometimes passed — the model guessed "mp3" without evidence. Determinism converted a lucky guess into an honest, consistent failure.
- Billing ("How Billing Works") — generation quality. At temperature 0 the small model deterministically echoes the source sentence — "This is the starting point for billing…" — without naming the page. Correct refusal-free retrieval, useless circular answer; the judge rightly fails it. Fix class: prompt work ("name the page/section you're citing") or a stronger generation model.
Spend the signal — one parameter, chosen by the eval
A trustworthy eval isn't the goal; it's the map. It just told us two of three failures are the
same class — facts that don't reach the top-4 retrieved chunks. The cheapest fix in that class is
one character, in retrieve():
def retrieve(question: str, k: int = 6): # was 4 — two misses said "not enough context"
Re-run. Then re-run again, because that's the habit now:
Results:
✓ 14 passed (87.50%)
✗ 2 failed (12.50%)
14/16 — 87.5% — twice, same two failures. Read what happened with each:
- TTS default — fixed. With six chunks the page stating the default reaches the context, and the service now answers "mp3" with evidence, every run — the lucky guess became a grounded answer.
- Subnets — survived. Even at
k=6the chunk with the stats block doesn't rank: this miss is semantic, not budgetary — the boundary-mangled text ("iated with subnets…") embeds poorly. Morekcan't fix what chunking broke; this one genuinely needs structure-aware chunking. - Billing — survived, as predicted. It's a generation-quality failure; no retrieval parameter was ever going to touch it.
The number is back to 87.5% — the same as the lucky run — but this time it reproduces, and every point of it has an explanation.
Figure 8. The final run: 14/16, the same two failures every time, regression PASS.
Read the final number honestly
39.5% → 44.7% → 75% → 87.5% (lucky) → 81.25% (honest) → 87.5% (earned). Until the very last step, the RAG barely changed — what changed was the measurement: assertions (substrings → judge), labels (curation gate), the judge's own config (truncation), determinism (temperature). Only then did one retrieval parameter, chosen by the eval, actually move quality. That's the last lesson of the series:
- Most "failures" were our eval lying to us — bad labels, brittle asserts, a silently truncated judge, a dice-rolling service. Read the rows before you touch the RAG.
- The honest number was lower than the lucky one — and worth more, because it reproduces. If your score changes when you re-run, you don't have a score yet.
- Fix the measurement first, then spend its signal. The same 87.5% appears twice in this arc; only the second one means something — it survives re-runs and every point has an explanation.
- The score is a trend instrument; the rows are the truth. 87.5% still isn't "the RAG's quality" — it's this dataset, this judge, this corpus. Pin what must never regress with deterministic asserts; let the judge track the rest.
What you built
- A Docker-first RAG service: scrape (sitemap → article extraction) → chunk → local
embeddings (fastembed/ONNX) → Chroma → WEC generation, behind
POST /askwith deterministic citations. - A real bug story: probed the negative case, caught a garbled GPU price (a storage rate welded
onto the GPU prepay option), root-caused it to the scraper's flattened structure, fixed ingestion
- hardened the prompt, and pinned it with a regression test that now passes.
- A CI-shaped eval of the live endpoint: gate-curated QA (15 verbatim-verified pairs out of 37 generated) + a judge you debugged like any other component — 39.5% → 87.5% reproducible, by fixing the measurement first (assertions, labels, judge config, determinism) and then spending its signal on one eval-chosen retrieval parameter. Two real, nameable failures remain — on purpose, with fix classes named.
- Full observability: every request a span tree in Langfuse — retrieval and generation separately debuggable.
Parts 1–4 were the toolkit. This is what the toolkit is for.
Troubleshooting
/ask returns nothing after re-indexing
If you drop and recreate the Chroma collection while the API is running, the service holds a stale collection handle and every query throws:
chromadb.errors.InvalidCollectionException: Collection 9b486740-… does not exist.
The running process cached the old collection's UUID. Restart the service
(docker compose restart rag-api) — or resolve the collection per-request instead of at startup.
Figure 9. Re-indexing invalidates open handles — a classic "works until you rebuild the index"
production trap.
Failed to send telemetry event … capture() takes 1 positional argument
Harmless chromadb/posthog version clash. Set ANONYMIZED_TELEMETRY: "False" in the environment
(already in our compose).
Judge failures with reason "No output" (and graderError: true)
Check the grading record's completion tokens: if it's exactly 1024 — Promptfoo's default
max_tokens — your thinking-model judge spent the whole budget reasoning and was cut off before
the verdict JSON. Set max_tokens: 4096 on the judge provider. Related tell: a small judge
returning reason: "string" is echoing the schema placeholder instead of grading — use a bigger
judge.
The judge eval is slow or verdicts look random
Reasoning judges are slow (~minutes for a dozen rows) and imperfect — expect occasional verdict flips between runs. Keep deterministic assertions for known bugs (they're free and never flip) and treat the judge score as a trend, not a truth. If a judge row hangs entirely, it's usually a transient network hiccup, not your config — retry before debugging.
Docker build fails with "no space left on device"
The image needs ~1 GB (chromadb + onnxruntime are the heavy layers). Check df -h / first and
docker system df for reclaimable space — but never prune blindly on a shared box: "reclaimable"
lies when other projects' containers are running.
What's next
The honest limitations are the roadmap, and each one traces to a failure you watched happen:
structure-aware chunking (the subnets miss survived k=6 — the boundary-mangled chunk embeds
poorly, so no retrieval budget rescues it), prompt or model work on generation (the billing
echo: force the model to name the page it's citing, or use a stronger generator), table-aware
extraction (the pricing strip is still fragment soup — the regression passes, but "$1.44/mo"
still bleeds near the GPU line; add a positive assert for the correct figure, not just
not-icontains for the wrong ones), and judge calibration (grade the judge against a handful
of human verdicts). Each one moves the pass rate for a reason you can name — which, after five
parts, is the whole point: you don't guess whether your AI works. You measure it, watch it, and
make it prove itself.