2 posts tagged with "langfuse"

++

Everything in this series was building to this. You can prove a model works (part 1), make its output machine-reliable (part 2), generate real test data (part 3), and observe production (part 4). Now we spend all four skills at once on the pattern behind almost every serious LLM product: RAG — retrieval-augmented generation.

We'll build a docs assistant: a containerized HTTP service that answers customer questions from WEC's own documentation. Not a notebook — a service, Docker-first, the shape you'd actually deploy. And because this series doesn't do happy-path demos: along the way our RAG hallucinates a GPU price, we root-cause it to our own scraper, fix it, and pin the fix with a regression test. Every command, number, and error below is from a real run.

+

A customer says your support bot promised them a refund policy that doesn't exist. Your feature made two LLM calls — classify, then reply. Which one invented it? If you can't answer that, your app is a black box — and this guide fixes exactly that.

You can now prove a model works (part 1), make its output machine-reliable (part 2), and generate a real test set to check it against (part 3). But all of that runs offline, in CI, on inputs you chose. Production doesn't play along.

This guide closes the gap. We'll self-host Langfuse — the open-source, self-hostable alternative to LangSmith — trace every real call, auto-score live traffic with an LLM judge, drill into the exact step that breaks, and feed failures back so your part-3 dataset gets stronger. Offline eval tells you it worked on your test set; this tells you it works in the wild.