5 posts tagged with "observability"

+

An agent that calls tools does two very different things: it reasons ("I should search the docs") and it acts (actually calls the tool). When something goes wrong, the first question is always which layer failed — did it think wrong, or did the doing break? A flat log can't answer that. A trace can.

In this tutorial you build a small tool-calling agent on WEC Inference, instrument it with Langfuse so every reasoning step and every tool call becomes an inspectable node, then debug two real failures from the trace tree — including the worst kind: a confident, wrong answer that never throws an error. Every command, error, and screenshot below is from a real run.

+

A customer says your support bot promised them a refund policy that doesn't exist. Your feature made two LLM calls — classify, then reply. Which one invented it? If you can't answer that, your app is a black box — and this guide fixes exactly that.

You can now prove a model works (part 1), make its output machine-reliable (part 2), and generate a real test set to check it against (part 3). But all of that runs offline, in CI, on inputs you chose. Production doesn't play along.

This guide closes the gap. We'll self-host Langfuse — the open-source, self-hostable alternative to LangSmith — trace every real call, auto-score live traffic with an LLM judge, drill into the exact step that breaks, and feed failures back so your part-3 dataset gets stronger. Offline eval tells you it worked on your test set; this tells you it works in the wild.

++

Your eval passes 100% — of the two test cases you typed by hand. Real users won't phrase things the way you did.

In part 1 and part 2 we built a real eval harness — assertions, schema validation, a model matrix. But two cases can't tell you if your app works; they can only tell you it didn't crash on two inputs.

This guide fixes that. The skill is producing a dataset you can actually trust: we use the WEC Inference API to generate labeled tickets, then validate and curate them — because generated labels are not automatically correct, and treating them as gospel just moves the bug. The result is real test data at scale. When you finally run it through the part-2 harness, coverage surfaces the failures — genuine misclassifications and debatable labels — that two hand-picked cases hide.

++

"Return only JSON" is one of the most common instructions in production LLM apps — and one of the least reliable. Models wrap JSON in markdown fences, add a friendly sentence, or (if they're reasoning models) narrate their entire thought process around it. Any of those breaks a strict JSON.parse, and your pipeline falls over.

In this guide we build a Promptfoo eval that makes WEC Inference API models classify support tickets into schema-validated JSON, then harden it against real-world messiness — and use it to pick a model you can actually trust. This is part 2 of the evals series (see part 1 for first-run setup). Everything here was run live against https://inference.wiline.com.

+

You wouldn't ship code without tests — but most teams ship LLM features on "looks good to me." Evaluations (evals) fix that: you define test cases and pass/fail criteria, then measure your model objectively — catching regressions, comparing models, and gating deploys.

In this guide you'll build a real eval harness with Promptfoo pointed at the WEC Inference API — the same OpenAI-compatible endpoint you call from your apps. Everything here was run live against https://inference.wiline.com; the outputs are real.