Stop hand-writing test cases: generate an eval dataset with the WEC API

++

0/6

🎯 Skill path0/6 earned

AI evals & observability

• 1Prove a model works
• 2Trustworthy JSON
• 3Real test data at scale
• 4Observe & score production
• 5RAG, end to end
• 🏆Catch regressions in CI

Finish this tutorial, then mark it complete below to earn the skill.

Your eval passes 100% — of the two test cases you typed by hand. Real users won't phrase things the way you did.

In part 1 and part 2 we built a real eval harness — assertions, schema validation, a model matrix. But two cases can't tell you if your app works; they can only tell you it didn't crash on two inputs.

This guide fixes that. The skill is producing a dataset you can actually trust: we use the WEC Inference API to generate labeled tickets, then validate and curate them — because generated labels are not automatically correct, and treating them as gospel just moves the bug. The result is real test data at scale. When you finally run it through the part-2 harness, coverage surfaces the failures — genuine misclassifications and debatable labels — that two hand-picked cases hide.

How it fits together

Each step feeds the next, ending in a dataset that plugs straight into the part-2 eval:

Prerequisites

• A WEC Inference API key — see Inference → API Keys.

• jq, curl, and Node.js 22+ (Promptfoo runs via npx).

• Your key exported and sanitized (a pasted key often carries invisible characters):

  export WEC_API_KEY='sk-your-key'
  export WEC_API_KEY=$(printf '%s' "$WEC_API_KEY" | LC_ALL=C tr -cd '[:print:]')

tip

If a call returns a 401 / "invalid API key", your key is expired or revoked — generate a fresh one in Inference → API Keys. See Troubleshooting.

Step 1 — Generate one labeled batch

Ask the model for tickets with their answers — the labels are what make it a dataset, not just inputs. We reuse the exact schema from part 2 (category + priority enums):

curl -s https://inference.wiline.com/v1/chat/completions \
  -H "Authorization: Bearer $WEC_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2.5-3B-Instruct",
    "temperature": 0.7,
    "messages": [{"role":"user","content":"Generate 8 diverse, realistic customer support tickets for a cloud hosting company. Return ONLY JSONL (one JSON object per line), each with keys: ticket (string), category (one of: billing, technical, account, other), priority (one of: low, medium, high). No markdown, no fences."}]
  }' | jq -r '.choices[0].message.content'

Output

{"ticket":"Our server went down unexpectedly and we're not sure what caused it.","category":"technical","priority":"high"}
{"ticket":"We need to upgrade our plan but the system is showing an error message when trying to change plans.","category":"account","priority":"medium"}
{"ticket":"We're having trouble accessing our database and can't seem to log in.","category":"technical","priority":"high"}
... 5 more lines ...

Figure 1. A single generation: 8 diverse tickets, each already carrying its category and priority label.

Clean JSONL, valid labels, real diversity — and it came back in seconds. We deliberately picked a small, non-reasoning model (Qwen2.5-3B-Instruct) for generation. Here's why that matters.

Why not a reasoning model for bulk generation?

Reasoning models return their chain-of-thought in a separate reasoning_content field — tokens you pay for but never use. Ask a reasoning model like Qwen3.5:9B for a few tickets and compare the hidden reasoning against the actual answer:

curl -s https://inference.wiline.com/v1/chat/completions \
  -H "Authorization: Bearer $WEC_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.5:9B","temperature":0.7,"messages":[{"role":"user","content":"Generate 3 realistic customer support tickets for a cloud hosting company. Return ONLY JSONL, keys: ticket, category, priority. No markdown, no fences."}]}' \
  | jq '{model, completion_tokens: .usage.completion_tokens, reasoning_chars: (.choices[0].message.reasoning_content|length), answer_chars: (.choices[0].message.content|length)}'

Output

{
  "model": "Qwen3.5:9B",
  "completion_tokens": 2000,
  "reasoning_chars": 7116,
  "answer_chars": 412
}

Figure 1b. For just 3 tickets, Qwen3.5:9B wrote 7,116 characters of hidden reasoning to produce 412 of answer — you pay for all ~2,000 completion tokens. (Counts vary per call; the lopsidedness doesn't.) That's why we generate with the small, non-reasoning Qwen2.5-3B — and it's the same reason those reasoning models stumble on structured output in Step 5's matrix.

Step 2 — Validate before you trust it

Never treat generated data as ground truth without checking it. Save a batch and validate three things: every line parses, every label is in-enum, and the classes are reasonably balanced.

curl -s https://inference.wiline.com/v1/chat/completions \
  -H "Authorization: Bearer $WEC_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"Qwen2.5-3B-Instruct","temperature":0.7,"messages":[{"role":"user","content":"Generate 8 diverse, realistic customer support tickets for a cloud hosting company. Return ONLY JSONL (one JSON object per line), each with keys: ticket (string), category (one of: billing, technical, account, other), priority (one of: low, medium, high). No markdown, no fences."}]}' \
  | jq -r '.choices[0].message.content' > tickets.jsonl

# 1) do all lines parse as JSON?
jq -e . tickets.jsonl > /dev/null && echo "all lines valid JSON" || echo "INVALID JSON present"

# 2) any out-of-enum labels? (no rows printed = all valid)
jq -c 'select((.category|IN("billing","technical","account","other")|not) or (.priority|IN("low","medium","high")|not))' tickets.jsonl

# 3) class distribution
jq -rs 'group_by(.category)[] | "\(.[0].category): \(length)"' tickets.jsonl

Output

all lines valid JSON
account: 2
billing: 2
other: 1
technical: 3

Figure 2. Validation output — every line parses, no out-of-enum labels, and a reasonable class spread.

jq gotcha

The intuitive ["billing",...] | index(.category) fails with "Cannot index array with string" — inside the pipe, . is the array, so .category tries to index it. Use .category | IN("billing", ...) instead. See Troubleshooting.

Step 3 — Scale up and dedupe

One batch of 8 isn't enough. Loop the call a few times (higher temperature for variety across batches), concatenate, then dedupe by ticket text:

for i in $(seq 1 5); do
  curl -s https://inference.wiline.com/v1/chat/completions \
    -H "Authorization: Bearer $WEC_API_KEY" -H "Content-Type: application/json" \
    -d '{"model":"Qwen2.5-3B-Instruct","temperature":0.9,"messages":[{"role":"user","content":"Generate 8 diverse, realistic customer support tickets for a cloud hosting company. Return ONLY JSONL (one JSON object per line), each with keys: ticket (string), category (one of: billing, technical, account, other), priority (one of: low, medium, high). No markdown, no fences."}]}' \
    | jq -r '.choices[0].message.content // empty'
  echo "batch $i done" >&2
done > tickets_raw.jsonl

# dedupe by ticket text
jq -sc 'unique_by(.ticket)[]' tickets_raw.jsonl > tickets_dataset.jsonl
echo "raw: $(wc -l < tickets_raw.jsonl)  →  deduped: $(wc -l < tickets_dataset.jsonl)"

Output

raw: 40  →  deduped: 40

Figure 3. Scaling up: 5 batches → 40 rows after exact-duplicate removal.

Zero exact duplicates across 5 batches — a good sign the model isn't just repeating itself.

caution

unique_by(.ticket) only catches identical text. Two differently-worded "server is down" tickets are semantic near-duplicates and will slip through — catching those needs embedding-based clustering, a more advanced step. For a starter dataset, exact dedup is fine.

Guard against silent failures

Note the // empty in the jq filter. If a call fails (revoked key, model timeout), the response has no content and .choices[0].message.content is null — a plain jq -r would then write the literal word null into your dataset, silently poisoning it. // empty drops those, so a failed call adds nothing instead of a junk row. Always check the row count afterward. See Troubleshooting.

Validate the full 40 the same way as Step 2 (jq -e ., the enum check, and both distributions). Ours came out balanced: technical 15 · account 10 · billing 10 · other 5; priority high 13 · medium 14 · low 13.

Skill unlocked 🏅

You now have a repeatable pipeline for real test data — 40 validated, deduped, labeled rows where you used to have two.

Step 4 — Convert to a Promptfoo dataset

Promptfoo reads test cases from a CSV: each column becomes a variable, and the special __expected column holds a per-row assertion. We map ticket → the prompt var and __expected → icontains:<the correct category>. jq @csv handles the commas and quotes inside ticket text:

{ echo 'ticket,__expected'; jq -r '[.ticket, ("icontains:" + .category)] | @csv' tickets_dataset.jsonl; } > tests.csv
wc -l tests.csv          # 41 = header + 40 rows

Step 5 — Put the dataset to work

The dataset is the deliverable — everything from here is what it unlocks. First payoff: a model comparison you can actually believe. On two hand-typed cases a ranking is noise; on 40 real, labeled rows it's a measurement. Add each WEC model as a provider and run the whole set:

cat > promptfooconfig.yaml <<'EOF'
description: "Ticket classifier — model matrix on the synthetic 40-row dataset"
providers:
  - id: openai:chat:Qwen2.5-3B-Instruct
    config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
  - id: openai:chat:Qwen3.5:9B
    config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
  - id: openai:chat:Qwen3.5-122B
    config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
  - |
    Classify this support ticket. Return ONLY JSON with keys:
    "category" (one of: billing, technical, account, other),
    "priority" (one of: low, medium, high),
    "summary" (string, max 12 words).
    Ticket: {{ticket}}
defaultTest:
  options:
    transform: |
      const m = output.match(/\{[\s\S]*?\}/);
      return m ? m[0] : output;
  assert:
    - type: is-json
      value:
        type: object
        required: [category, priority, summary]
        properties:
          category: { type: string, enum: [billing, technical, account, other] }
          priority: { type: string, enum: [low, medium, high] }
          summary: { type: string }
tests: file://tests.csv
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml

The result is the opposite of what "bigger is better" would predict:

Model	Accuracy	Completion tokens / ticket
Qwen2.5-3B-Instruct (non-reasoning)	85% (34/40)	~30
Qwen3.5:9B (reasoning)	47.5% (19/40)	~965
Qwen3.5-122B (reasoning)	15% (6/40)	~1,013

Figure 4. Same dataset, three models — the small non-reasoning model wins by a mile.

The small, non-reasoning model wins decisively — and the bigger the reasoning model, the worse it does. The reasoning models aren't dumber at classifying; they bury the JSON in their chain-of-thought. Qwen3.5:9B leaks Thinking: … into the output; Qwen3.5-122B often returns an empty answer entirely — all ~1,000 tokens spent reasoning, nothing parseable left. They also burn ~30× more tokens per ticket, which is why the full matrix took 12m 31s, almost all of it spent waiting on the two big models. For structured output, thinking out loud is a liability, not an advantage — a counterintuitive finding you'd never get from a leaderboard.

But notice what actually made that finding possible: the dataset. On two hand-typed cases those percentages would be coin-flips; 40 labeled rows are what turn "which model?" from a guess into a measurement. The comparison is the reward for building the data — not a substitute for it.

Skill unlocked 🏅

You can now pick a model with evidence instead of vibes — something no leaderboard can do for you, because it doesn't know your task.

(The transform grabs the first {…} block with a non-greedy *? match — cheap insurance for models that wrap JSON in prose, exactly like part 2.)

We score category here — priority is your next assertion

The __expected column grades only the predicted category against the label. The dataset also carries a priority label we don't score yet. To grade both fields at once, add a javascript assertion that parses the JSON output and compares priority too — the exact same pattern, one more check. Scoring more of what you generated is the cheapest way to make an eval stricter.

Step 6 — Dig into the winner's failures

Take the winner — Qwen2.5-3B at 85% (34/40) — and pull its 6 failing tickets, comparing each to its label. (__expected isn't stored as a variable — Promptfoo consumes it as the assertion — so read the ground truth from the source dataset.) Because the data is freshly generated, your exact failures will differ, but they consistently split into two recognizable kinds — and neither is just "the model is dumb". Representative examples:

Figure 5. The web report makes it easy to scan which rows failed and open each one.

Ticket	Label	Why it's tricky
"educational discounts for startups…"	other	Debatable — it's about pricing, so billing is defensible
"grant a junior dev IAM access to S3…"	account	Debatable — access control vs infrastructure
"feedback on the API documentation…"	other	Debatable — feedback vs technical
"update the card and add a user…"	account	Multi-intent — genuinely two categories at once

The rest are ordinary misclassifications. The real, day-to-day lessons:

• Your synthetic labels are not gospel. A chunk of "failures" are label disputes — review and fix them, or your eval measures the wrong thing.
• Single-label classification breaks on multi-intent tickets. Real inputs aren't always one category.
• An 85% headline hides both. You only learn why by reading the failing rows — which is the entire reason to eval on a real dataset instead of two examples.

What we built

• The core skill: a repeatable pipeline to generate, validate, dedupe, and version an eval dataset from the WEC API — real test data instead of two hand-typed cases.
• The discipline that makes it trustworthy: generated labels aren't gospel. You validate enums and balance, and you review the disputes — a chunk of "failures" are your own labels being wrong.
• Proof it matters: coverage surfaces what two cases hide — genuine misclassifications and debatable, multi-intent tickets you'd never think to type by hand.
• A bonus the data unlocks: a trustworthy model comparison — here a 3B non-reasoning model beat a 122B reasoning one for structured output, the opposite of "bigger is better."

Commit tickets_dataset.jsonl and tests.csv alongside your eval config, and regenerate/grow the dataset as your product changes.

Troubleshooting

401 / invalid API key

Output

{ "error": { "message": "Authentication Error, invalid API key", "code": "401" } }

The key is expired, revoked, or malformed. Generate a fresh one in Inference → API Keys, re-export it, and re-sanitize (tr -cd '[:print:]') in case the paste carried an invisible character.

jq: "Cannot index array with string"

This comes from ["billing", ...] | index(.category) — inside the pipe, . is the array, so .category indexes the array. Use the IN() idiom instead:

jq -c 'select((.category|IN("billing","technical","account","other")|not))' tickets.jsonl

Figure 6. The index(.field) mistake — inside the pipe, jq tries to index the array with a string.

Your dataset has rows that are just null

A null row means a generation call failed but the loop still wrote its (empty) result. Usual causes: an expired/revoked API key, or the model timing out. Fixes: extract with .choices[0].message.content // empty so a failed call writes nothing instead of null; re-check wc -l after every generation run; and if calls hang, add a per-request timeout (curl -m 60) and consider a lighter, faster model for bulk generation.

__expected is null in the results JSON

That's expected — Promptfoo consumes the __expected CSV column as the row's assertion, so it isn't kept as a variable. To inspect ground-truth labels for failing rows, read them from your source dataset (tickets_dataset.jsonl) rather than the results file.

🎯

Finished this tutorial?

Mark it complete to earn Real test data at scale on your skill path.

What's next

You now have a real dataset — but it runs in CI, offline. The next step is watching your app in production: self-hosting Langfuse to trace every call, track latency and cost, and run evals (using this dataset) against live traffic. That's where offline testing becomes real observability — and it's the next post in the series.