7 posts tagged with "evals"

+

An agent that calls tools does two very different things: it reasons ("I should search the docs") and it acts (actually calls the tool). When something goes wrong, the first question is always which layer failed — did it think wrong, or did the doing break? A flat log can't answer that. A trace can.

In this tutorial you build a small tool-calling agent on WEC Inference, instrument it with Langfuse so every reasoning step and every tool call becomes an inspectable node, then debug two real failures from the trace tree — including the worst kind: a confident, wrong answer that never throws an error. Every command, error, and screenshot below is from a real run.

++

Our GPT-5.6 token-economics analysis ended with a challenge: benchmark promises are a hypothesis, not a reason — run the eval on your workload before you believe a "fewer tokens per task" claim. This tutorial is us taking our own advice.

We take the RAG docs assistant from part 5, wrap it in a DeepEval regression suite — containerized, judged by gemma4 on the WEC Inference API, zero OpenAI dependency — and then use that suite to answer a genuinely open question: should we swap the service's generation model? Qwen2.5-3B-Instruct (current) vs gemma4 (candidate), pass rate and tokens per task, measured head to head.

Spoiler: the eval saved us from a pointless migration. And along the way we hit four real production failures — a disk-full death spiral, a startup race condition, a Docker Compose variable trap that silently evaluated the wrong model, and the fix that makes that class of bug impossible. Every command, number, and error below is from a real run.

++

Everything in this series was building to this. You can prove a model works (part 1), make its output machine-reliable (part 2), generate real test data (part 3), and observe production (part 4). Now we spend all four skills at once on the pattern behind almost every serious LLM product: RAG — retrieval-augmented generation.

We'll build a docs assistant: a containerized HTTP service that answers customer questions from WEC's own documentation. Not a notebook — a service, Docker-first, the shape you'd actually deploy. And because this series doesn't do happy-path demos: along the way our RAG hallucinates a GPU price, we root-cause it to our own scraper, fix it, and pin the fix with a regression test. Every command, number, and error below is from a real run.

+

A customer says your support bot promised them a refund policy that doesn't exist. Your feature made two LLM calls — classify, then reply. Which one invented it? If you can't answer that, your app is a black box — and this guide fixes exactly that.

You can now prove a model works (part 1), make its output machine-reliable (part 2), and generate a real test set to check it against (part 3). But all of that runs offline, in CI, on inputs you chose. Production doesn't play along.

This guide closes the gap. We'll self-host Langfuse — the open-source, self-hostable alternative to LangSmith — trace every real call, auto-score live traffic with an LLM judge, drill into the exact step that breaks, and feed failures back so your part-3 dataset gets stronger. Offline eval tells you it worked on your test set; this tells you it works in the wild.

++

Your eval passes 100% — of the two test cases you typed by hand. Real users won't phrase things the way you did.

In part 1 and part 2 we built a real eval harness — assertions, schema validation, a model matrix. But two cases can't tell you if your app works; they can only tell you it didn't crash on two inputs.

This guide fixes that. The skill is producing a dataset you can actually trust: we use the WEC Inference API to generate labeled tickets, then validate and curate them — because generated labels are not automatically correct, and treating them as gospel just moves the bug. The result is real test data at scale. When you finally run it through the part-2 harness, coverage surfaces the failures — genuine misclassifications and debatable labels — that two hand-picked cases hide.

++

"Return only JSON" is one of the most common instructions in production LLM apps — and one of the least reliable. Models wrap JSON in markdown fences, add a friendly sentence, or (if they're reasoning models) narrate their entire thought process around it. Any of those breaks a strict JSON.parse, and your pipeline falls over.

In this guide we build a Promptfoo eval that makes WEC Inference API models classify support tickets into schema-validated JSON, then harden it against real-world messiness — and use it to pick a model you can actually trust. This is part 2 of the evals series (see part 1 for first-run setup). Everything here was run live against https://inference.wiline.com.

+

You wouldn't ship code without tests — but most teams ship LLM features on "looks good to me." Evaluations (evals) fix that: you define test cases and pass/fail criteria, then measure your model objectively — catching regressions, comparing models, and gating deploys.

In this guide you'll build a real eval harness with Promptfoo pointed at the WEC Inference API — the same OpenAI-compatible endpoint you call from your apps. Everything here was run live against https://inference.wiline.com; the outputs are real.