2 posts tagged with "evals"

++

"Return only JSON" is one of the most common instructions in production LLM apps — and one of the least reliable. Models wrap JSON in markdown fences, add a friendly sentence, or (if they're reasoning models) narrate their entire thought process around it. Any of those breaks a strict JSON.parse, and your pipeline falls over.

In this guide we build a Promptfoo eval that makes WEC Inference API models classify support tickets into schema-validated JSON, then harden it against real-world messiness — and use it to pick a model you can actually trust. This is part 2 of the evals series (see part 1 for first-run setup). Everything here was run live against https://inference.wiline.com.

+

You wouldn't ship code without tests — but most teams ship LLM features on "looks good to me." Evaluations (evals) fix that: you define test cases and pass/fail criteria, then measure your model objectively — catching regressions, comparing models, and gating deploys.

In this guide you'll build a real eval harness with Promptfoo pointed at the WEC Inference API — the same OpenAI-compatible endpoint you call from your apps. Everything here was run live against https://inference.wiline.com; the outputs are real.