Evaluate your models with Promptfoo on the WEC Inference API

+

You wouldn't ship code without tests — but most teams ship LLM features on "looks good to me." Evaluations (evals) fix that: you define test cases and pass/fail criteria, then measure your model objectively — catching regressions, comparing models, and gating deploys.

In this guide you'll build a real eval harness with Promptfoo pointed at the WEC Inference API — the same OpenAI-compatible endpoint you call from your apps. Everything here was run live against https://inference.wiline.com; the outputs are real.

Prerequisites

• A WEC (WiLine Edge Cloud) account and an Inference API key — see Inference → API Keys.

• Node.js 22+. Promptfoo treats Node 20 as end-of-life; with nvm:

  nvm install 24      # 24 LTS preferred
  nvm use 24
  node -v             # v24.x

We'll run Promptfoo with npx — nothing to install globally.

Figure 1. Create a key under Inference → API Keys, then copy it.

Put your key in the shell:

export WEC_API_KEY='sk-your-key-here'

tip

If your first eval errors with Cannot convert argument to a ByteString, see Troubleshooting #1 below.

Step 1 — See which models you have

The API is OpenAI-compatible, so the model list is a plain GET /v1/models:

curl -s https://inference.wiline.com/v1/models \
  -H "Authorization: Bearer $WEC_API_KEY" | jq -r '.data[].id'

Output

zai-org/GLM-5.2
gemma4
Qwen2.5-3B-Instruct
qwen3.5:9B
whisper-large-v3
whisper-medium
whisper-base

The whisper-* models are speech-to-text; the rest are chat models. We'll evaluate the chat models, starting with zai-org/GLM-5.2.

Step 2 — Your first eval

Create a config. apiBaseUrl points Promptfoo at WEC; apiKeyEnvar tells it which env var holds your key:

cat > promptfooconfig.yaml <<'EOF'
description: "WEC Inference API eval"
providers:
  - id: openai:chat:zai-org/GLM-5.2
    config:
      apiBaseUrl: https://inference.wiline.com/v1
      apiKeyEnvar: WEC_API_KEY
      temperature: 0
prompts:
  - "Answer in a single word. What is the capital of {{country}}?"
tests:
  - vars: { country: France }
    assert: [{ type: icontains, value: Paris }]
  - vars: { country: Japan }
    assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml

Figure 4. Your first passing eval against the WEC Inference API.

icontains is a deterministic assertion — a case-insensitive substring check. Cheap, fast, and perfect for facts with a known answer.

Step 3 — The visual report

npx -y promptfoo@latest view

This opens a local web UI with every prompt, output, and pass/fail — far easier to scan than the terminal once you have more than a handful of cases.

Figure 5. promptfoo view — the same results in the browser.

Step 4 — Add a second model

Adding a provider is one extra block. Here we add Qwen2.5-3B-Instruct alongside GLM-5.2 and run the same two test cases against both:

cat > promptfooconfig.yaml <<'EOF'
description: "WEC Inference API eval — model comparison"
providers:
  - id: openai:chat:zai-org/GLM-5.2
    config:
      apiBaseUrl: https://inference.wiline.com/v1
      apiKeyEnvar: WEC_API_KEY
      temperature: 0
  - id: openai:chat:Qwen2.5-3B-Instruct
    config:
      apiBaseUrl: https://inference.wiline.com/v1
      apiKeyEnvar: WEC_API_KEY
      temperature: 0
prompts:
  - "Answer in a single word. What is the capital of {{country}}?"
tests:
  - vars: { country: France }
    assert: [{ type: icontains, value: Paris }]
  - vars: { country: Japan }
    assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml

Figure 6. Both models pass. GLM-5.2's results are served from cache (0 new requests); only Qwen2.5 makes live calls.

Step 5 — Grade quality, not substrings (LLM rubric)

Substring checks can't judge an open-ended answer. A model-graded llm-rubric assertion uses a model to score the output against criteria — and the grader can run on WEC too, so you need no external provider:

cat > promptfooconfig.yaml <<'EOF'
description: "WEC Inference API eval — model-graded (LLM rubric)"
providers:
  - id: openai:chat:zai-org/GLM-5.2
    config:
      apiBaseUrl: https://inference.wiline.com/v1
      apiKeyEnvar: WEC_API_KEY
      temperature: 0
defaultTest:
  options:
    provider:                       # the grader model
      id: openai:chat:zai-org/GLM-5.2
      config:
        apiBaseUrl: https://inference.wiline.com/v1
        apiKeyEnvar: WEC_API_KEY
prompts:
  - "Explain what a {{topic}} is in two sentences, for a beginner."
tests:
  - vars: { topic: VPC }
    assert:
      - type: llm-rubric
        value: "Describes a VPC as a private, isolated virtual network in the cloud, and is understandable to a beginner."
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml

Figure 7. The run shows two token lines — Eval (75) and Grading (350) — because the rubric check is itself a model call.

Model-graded checks are powerful, but they aren't free.

Step 6 — Guardrails: latency

A correct answer that takes 30 seconds is still a production problem. Add a latency guardrail to defaultTest (it applies to every case). Always run with --no-cache — cached responses report ~0 ms and would pass the gate for free:

cat > promptfooconfig.yaml <<'EOF'
description: "WEC Inference API eval — latency guardrail"
providers:
  - id: openai:chat:zai-org/GLM-5.2
    config:
      apiBaseUrl: https://inference.wiline.com/v1
      apiKeyEnvar: WEC_API_KEY
      temperature: 0
defaultTest:
  assert:
    - type: latency
      threshold: 5000        # fail anything slower than 5 s
prompts:
  - "Answer in a single word. What is the capital of {{country}}?"
tests:
  - vars: { country: France }
    assert: [{ type: icontains, value: Paris }]
  - vars: { country: Japan }
    assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache

This failed — the answers were correct (Paris, Tokyo) but both calls took ~33 s, tripping the 5 s gate:

Figure 8. The guardrail fires: correct answers, wrong latency. Duration: 33s.

Bump the threshold to something realistic and re-run:

sed -i 's/threshold: 5000/threshold: 60000/' promptfooconfig.yaml
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache

Figure 9. With a 60 s threshold the same calls pass easily — Duration: 2s.

Inference latency varies — calibrate before gating

Back-to-back calls to the same model ranged from ~2 s to ~33 s on the same day. Run a few --no-cache passes to get a realistic baseline before locking in your threshold.

Step 7 — Guardrails: latency + token cap

The cost assertion doesn't work against WEC (see Troubleshooting #2). Use latency + max_tokens together as your production guardrail instead:

cat > promptfooconfig.yaml <<'EOF'
description: "WEC Inference API eval — guardrails (latency + token cap)"
providers:
  - id: openai:chat:zai-org/GLM-5.2
    config:
      apiBaseUrl: https://inference.wiline.com/v1
      apiKeyEnvar: WEC_API_KEY
      temperature: 0
      max_tokens: 16
defaultTest:
  assert:
    - type: latency
      threshold: 30000
prompts:
  - "Answer in a single word. What is the capital of {{country}}?"
tests:
  - vars: { country: France }
    assert: [{ type: icontains, value: Paris }]
  - vars: { country: Japan }
    assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache

Figure 11. With a 30 s threshold and a 16-token cap, both cases pass.

Step 8 — Validate structured output (JSON schema)

Real apps want JSON, not prose. is-json checks the output is valid JSON and matches a schema in one assertion:

cat > promptfooconfig.yaml <<'EOF'
description: "WEC Inference API eval — JSON schema validation"
providers:
  - id: openai:chat:zai-org/GLM-5.2
    config:
      apiBaseUrl: https://inference.wiline.com/v1
      apiKeyEnvar: WEC_API_KEY
      temperature: 0
prompts:
  - 'Return ONLY a JSON object (no markdown, no prose) with keys "capital" (string) and "population_millions" (number) for the country {{country}}.'
defaultTest:
  assert:
    - type: is-json
      value:
        type: object
        required: [capital, population_millions]
        properties:
          capital: { type: string }
          population_millions: { type: number }
tests:
  - vars: { country: France }
    assert: [{ type: icontains, value: Paris }]
  - vars: { country: Japan }
    assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml

Figure 12. GLM-5.2 returns compact, schema-valid JSON on the first try.

Step 9 — The payoff: compare every WEC model

Here's what you can't get anywhere else — the same strict-JSON test across all the chat models on your account:

cat > promptfooconfig.yaml <<'EOF'
description: "WEC Inference API eval — model matrix"
providers:
  - id: openai:chat:zai-org/GLM-5.2
    config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
  - id: openai:chat:gemma4
    config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
  - id: openai:chat:Qwen2.5-3B-Instruct
    config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
  - id: openai:chat:qwen3.5:9B
    config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
  - 'Return ONLY a JSON object (no markdown, no prose) with keys "capital" (string) and "population_millions" (number) for the country {{country}}.'
defaultTest:
  assert:
    - type: is-json
      value:
        type: object
        required: [capital, population_millions]
        properties:
          capital: { type: string }
          population_millions: { type: number }
tests:
  - vars: { country: France }
    assert: [{ type: icontains, value: Paris }]
  - vars: { country: Japan }
    assert: [{ type: icontains, value: Tokyo }]
  - vars: { country: Brazil }
    assert: [{ type: icontains, value: Bras }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml

Figure 13a. 12 test cases across 4 models — 9 pass, 3 fail. qwen3.5:9B emits its chain-of-thought instead of JSON.

Figure 13b. The token breakdown tells the real story: the reasoning model cost 13× more for the same task.

Model	Strict JSON?	Notes	Tokens (3 calls)
zai-org/GLM-5.2	✅ all pass	compact JSON	177
gemma4	✅ all pass	integer populations	207
Qwen2.5-3B-Instruct	✅ all pass	pretty-printed, precise	262
qwen3.5:9B	❌ all fail	reasoning model — emits its "thinking" instead of raw JSON	2,650

Two lessons fall out immediately:

• qwen3.5:9B is a reasoning model. It "thinks out loud," so it breaks strict JSON unless you strip the reasoning or use a response-format constraint. Picking it for structured output would silently fail.
• Cost varies ~13×. The reasoning model burned 2,650 tokens for the same task the others did in ~200. For a JSON extraction job, the smaller models are both correct and far cheaper.

That's a model-selection decision you can now defend with data — for your workloads, not a generic leaderboard.

Step 10 — Make it a gate

Promptfoo exits non-zero (100) when any test fails. Verify this first:

npx -y promptfoo@latest eval -c promptfooconfig.yaml > /dev/null 2>&1; echo "exit code: $?"

Figure 14. Exit code 100 confirms Promptfoo signals failure reliably — exactly what CI needs.

Wire it up as a gate:

npx -y promptfoo@latest eval -c promptfooconfig.yaml \
  && echo "✅ eval passed" \
  || { echo "❌ eval failed — blocking"; exit 1; }

Figure 15a. The gate re-runs the matrix eval from cache — same 9/3 result.

Figure 15b. Promptfoo exits non-zero, the || branch fires, and the gate blocks.

tip

Never run the exit 1 variant in your interactive shell — see Troubleshooting #3.

In CI, you don't even need the exit 1 — the non-zero exit fails the job. On GitLab:

eval:
  image: node:22
  script:
    - npx -y promptfoo@latest eval -c promptfooconfig.yaml
  # WEC_API_KEY is set as a masked CI/CD variable — never commit it.

Now a prompt change that quietly breaks JSON output, or a model swap that doubles latency, fails the pipeline before it reaches users.

Troubleshooting (real errors)

1. ByteString error on first eval

Output

API call error: TypeError: Cannot convert argument to a ByteString because
the character at index 32 has a value of 8232 which is greater than 255.

Figure 2. Every API call errors before reaching the server — the key never even leaves the machine.

Cause: The pasted key contains an invisible Unicode character — U+2028 (line separator). Some terminals and clipboard managers insert it silently when copying from certain sources.

Fix: Strip non-printable characters from the key:

export WEC_API_KEY=$(printf '%s' "$WEC_API_KEY" | LC_ALL=C tr -cd '[:print:]')
printf 'len=%s\n' "$(printf %s "$WEC_API_KEY" | wc -c)"
# len=25

Figure 3. After stripping, the key length confirms no extra characters remain. Re-run the eval — it passes.

---

2. Cost assertion errors immediately

Output

Error: Cost assertion does not support providers that do not return cost

Figure 10. The WEC Inference API doesn't return a cost field in its responses.

Cause: Promptfoo calculates cost by looking up a price-per-token for the model in its built-in price list and multiplying by the tokens used. It has prices for the big hosted models but not for WEC's self-hosted model names, so it has no price to apply and the assertion fails. (Most APIs, including OpenAI, don't return a dollar cost in the response — tools compute it.)

Fix: Skip the cost assertion entirely. Track spend from the Total Tokens line Promptfoo prints at the end of every run, and use max_tokens in the provider config to cap per-call token usage.

---

3. exit 1 closes your SSH session

Output

❌ eval failed — blocking
logout
Connection to 10.80.4.212 closed.

Cause: Running || { echo "❌ eval failed — blocking"; exit 1; } directly in an interactive shell exits the shell process itself — which, over SSH, closes the connection.

Fix: Put the gate in a non-interactive context: a Makefile target, an npm script, a git pre-commit hook, or a CI job. In those environments exit 1 fails the job without touching your terminal. If you need to test the gate locally, use || echo "❌ eval failed" (without exit 1) to see the output safely.

---

What's next

You've gone from "looks good to me" to a real, automatable eval harness on WEC — with deterministic checks, guardrails, schema validation, model-graded rubrics, a model matrix, and a CI gate.

Coming up in this series:

• Trustworthy JSON: schema-validate your model's structured output — now available (part 2): recover fenced and reasoning-wrapped output with transforms, compare models for JSON reliability, and gate it all in CI.
• Model-graded evals done right — calibrating llm-rubric, factuality scoring, and multi-judge checks so your grader is trustworthy.
• Catching hallucinations — pairing these techniques with WEC's RAG hallucination evaluation.
• Evaluating audio — scoring Whisper transcription accuracy on the WEC Inference API.