![\"LiteLLM\"](\"https://wec.wiline.com/docs/assets/images/litellm-wordmark-86d6f0432720b27534e05bc579dfc29c.png\")
++Here's how it usually goes. One app needs a model, so you paste the API key into\nits .env. Then a second app needs one. Then a script. Six months later the same\nkey is in five places, nobody remembers which of them is still running, and you\ncan't rotate it without breaking something you'll only find out about when it\nbreaks.
A gateway is the boring fix. One endpoint in front of every model, one place that\nholds the real credential, and a scoped key per app that you can revoke on its own.\nThis post deploys one on a WEC Instance and points it at the WEC Inference API.
\n\nThe host already runs five other stacks, which matters ā this is the normal case,\nnot a clean VM. Before adding anything, check what's listening and what's free:
\ndocker ps --format '{{.Names}}\\t{{.Ports}}'
sudo ss -tlnp | grep -E ':(4000|5432|5433)\\b'
langfuse-postgres-1 127.0.0.1:5433->5432/tcp
langfuse-clickhouse-1 127.0.0.1:8123->8123/tcp, 127.0.0.1:9000->9000/tcp
openclaw-caddy-1 100.87.239.229:80->80/tcp, 100.87.239.229:443->443/tcp
...
LISTEN 0 244 127.0.0.1:5432 users:((\"postgres\",pid=892))
LISTEN 0 4096 127.0.0.1:5433 users:((\"docker-proxy\"))
Port 4000 is free. Postgres 5432 belongs to the host and 5433 to Langfuse, so the\ngateway's database gets neither ā it won't publish a port at all.
\nThen confirm the backend answers before putting anything in front of it:
\ncurl -s https://inference.wiline.com/v1/models \\
-H \"Authorization: Bearer $WEC_API_KEY\" | jq -r '.data[].id'
whisper-large-v3
gemma4
wiline-coding
zai-org/GLM-5.2
Qwen2.5-3B-Instruct
Qwen3.5:9B
Qwen3.5-122B
kokoro
Llama3.1-8B-Instruct
bge-m3
Qwen3.5-9B
wiline-auto
wiline-cost
Thirteen models on one key. That's the thing we're about to stop handing out.
\nTwo containers: the gateway and a Postgres for its own state. Make a directory and\nwrite the compose file:
\nmkdir -p ~/llm-gateway && cd ~/llm-gateway
services:
litellm:
image: ghcr.io/berriai/litellm:main-stable
container_name: llm-gateway
restart: unless-stopped
ports:
- \"127.0.0.1:4000:4000\"
volumes:
- ./config.yaml:/app/config.yaml:ro
command: [\"--config\", \"/app/config.yaml\", \"--port\", \"4000\"]
env_file: .env
depends_on:
db:
condition: service_healthy
db:
image: postgres:17
container_name: llm-gateway-db
restart: unless-stopped
user: \"999:999\"
environment:
POSTGRES_USER: llmproxy
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: litellm
volumes:
- gateway-db:/var/lib/postgresql/data
healthcheck:
test: [\"CMD-SHELL\", \"pg_isready -U llmproxy -d litellm\"]
interval: 5s
timeout: 5s
retries: 10
volumes:
gateway-db:
Four of those lines are doing real work:
\n127.0.0.1:4000:4000 keeps the gateway off the public internet. It holds every\nmodel credential you own, so publishing it to 0.0.0.0 would put all of them\nbehind a single shared password. Part 1 of the hardening\nseries is the long version of why\nthat's a bad trade.
The database has no ports: at all. Nothing outside the compose network needs to\nreach it, and 5432 was taken anyway.
user: \"999:999\" runs Postgres as its built-in non-root user from the first boot.\nPart 2 covers what that buys you\nand how to find the right UID for an image instead of guessing.
condition: service_healthy matters more than it looks. The gateway runs database\nmigrations on startup ā without it, it races Postgres and crash-loops.
model_list:
- model_name: qwen-small
litellm_params:
model: openai/Qwen2.5-3B-Instruct
api_base: https://inference.wiline.com/v1
api_key: os.environ/WEC_API_KEY
- model_name: qwen-mid
litellm_params:
model: openai/Qwen3.5-9B
api_base: https://inference.wiline.com/v1
api_key: os.environ/WEC_API_KEY
- model_name: qwen-large
litellm_params:
model: openai/Qwen3.5-122B
api_base: https://inference.wiline.com/v1
api_key: os.environ/WEC_API_KEY
- model_name: embeddings
litellm_params:
model: openai/bge-m3
api_base: https://inference.wiline.com/v1
api_key: os.environ/WEC_API_KEY
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
store_model_in_db: true
litellm_settings:
drop_params: true
model_name is the alias your apps ask for. model: is what the gateway actually\ncalls. That indirection is most of the value ā swap qwen-mid to point somewhere\nelse next month and not one client changes.
The openai/ prefix says \"speak the OpenAI protocol to a custom api_base.\" The\nWEC Inference API is OpenAI-compatible, so that's all it takes.
Generate the passwords rather than inventing them:
\nPGPW=$(openssl rand -hex 16)
MK=\"sk-$(openssl rand -hex 20)\"
cat > .env <<EOF
POSTGRES_PASSWORD=${PGPW}
DATABASE_URL=postgresql://llmproxy:${PGPW}@db:5432/litellm
LITELLM_MASTER_KEY=${MK}
WEC_API_KEY=your-wec-key-here
EOF
chmod 600 .env
Then put your real WEC key in place of your-wec-key-here. The master key is the\ngateway's root credential ā it can create keys, delete keys and reach every model,\nso it goes in your password manager and nowhere else.
docker compose up -d
ā Image ghcr.io/berriai/litellm:main-stable Pulled 58.2s
ā Network llm-gateway_default Created 0.1s
ā Volume llm-gateway_gateway-db Created 0.1s
ā Container llm-gateway-db Healthy 8.8s
ā Container llm-gateway Created 0.2s
![\"docker](\"https://wec.wiline.com/docs/assets/images/gw-compose-up-e741e826f043d86aa7f1a7214f71fbcb.png\")
\nFigure 1. One pull, two containers, a network and a volume.
Created rather than Started on that last line is normal for the compose output,\nbut check anyway ā and check what the image cost you in disk while you're there:
docker compose ps -a && echo && df -h /
NAME IMAGE SERVICE STATUS PORTS
llm-gateway ghcr.io/berriai/litellm:main-stable litellm Up 3 minutes 127.0.0.1:4000->4000/tcp
llm-gateway-db postgres:17 db Up 3 minutes (healthy) 5432/tcp
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 58G 54G 4.2G 93% /
Note the database's PORTS column ā 5432/tcp with no host binding in front of\nit. The gateway image is 1.16GB, which on a box already running five stacks is not\nnothing. Check you have the room before you start, not after.
Two lines in the startup log are worth reading, because both look worse than they are:
\ndocker compose logs litellm --tail 40
ā
Migration diff applied successfully
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)
register_model: model=openai/Qwen2.5-3B-Instruct not in built-in cost map and no
prefix/region variant matched; cache cost fields will default to 0.
![\"Container](\"https://wec.wiline.com/docs/assets/images/gw-ps-logs-ed50299979281a4fb3dfd6ec9c89d189.png\")
\nFigure 2. Migrations applied, application startup complete, and one cost-map\nwarning per model.
0.0.0.0:4000 is the bind inside the container. The host publish is still\n127.0.0.1, which is what actually gates access. And the cost-map warning is\nnarrower than it reads ā it's about cache pricing specifically. Regular token\npricing still happens, which turns out to matter later.
set -a && . ./.env && set +a
curl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: gateway works\"}],\"max_tokens\":20}' | jq
\"finish_reason\": \"length\",
\"message\": {
\"reasoning_content\": \"Thinking Process:\\n\\n1. **Analyze the Request:**\\n * Input: \\\"\",
\"content\": null
}
\"usage\": {\"completion_tokens\": 20, \"prompt_tokens\": 16, \"total_tokens\": 36}
![\"Raw](\"https://wec.wiline.com/docs/assets/images/gw-reasoning-null-2d60fce2efcb8e175da27120082ac585.png\")
\nFigure 3. A successful call with nothing in it.
content: null. The call worked ā it routed, it hit the backend, it counted tokens\nā but there's no answer in it.
Qwen3.5-9B is a reasoning model. It spent all twenty tokens thinking and had none\nleft to speak with. The obvious move is to give it more room:
curl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: gateway works\"}],\"max_tokens\":200}' | jq '.choices[0].message.content, .usage'
null
{
\"completion_tokens\": 200,
\"prompt_tokens\": 16,
\"total_tokens\": 216
}
Two hundred tokens, still nothing. And this is the part worth pausing on: an earlier\nrun of that exact command did answer, at 192 tokens. Same model, same prompt,\ndifferent amount of thinking. There's no max_tokens you can set that reliably\nbuys you an answer, because the reasoning length isn't fixed.
Now the same prompt through the small model, at the original twenty tokens:
\ncurl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-small\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: gateway works\"}],\"max_tokens\":20}' | jq '.choices[0].message.content, .usage'
\"gateway works\"
{
\"completion_tokens\": 3,
\"prompt_tokens\": 35,
\"total_tokens\": 38
}
![\"Reasoning](\"https://wec.wiline.com/docs/assets/images/gw-token-contrast-922bab5aa94a6417e02dbc0d6fe0c874.png\")
\nFigure 4. Two hundred tokens and no answer, against three tokens and the answer.
Nothing is broken here. The reasoning model is doing exactly what it's built to do,\non a question that didn't need it. The fix isn't a bigger token budget ā it's not\nsending trivial work to that model in the first place. Which is the entire argument\nfor routing, and the subject of the next post.
\nYou can put a gateway in front of several models, give each one an alias your apps\ncall instead of a provider model ID, and read a response well enough to tell a\nreasoning model from a plain one.
A gateway that holds every credential you own should refuse anyone without a key.\nWorth checking rather than assuming:
\ncurl -s -o /dev/null -w \"no key: %{http_code}\\n\" http://127.0.0.1:4000/v1/chat/completions \\
-H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'
curl -s -o /dev/null -w \"wrong key: %{http_code}\\n\" http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer sk-not-a-real-key\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'
curl -s -o /dev/null -w \"master key: %{http_code}\\n\" http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-mid\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":5}'
no key: 401
wrong key: 401
master key: 200
![\"Three](\"https://wec.wiline.com/docs/assets/images/gw-auth-401-a9d9d89ff8432a6ec7a29486800ea82c.png\")
\nFigure 5. Anonymous and forged both rejected, the real key through.
The gateway ships a web interface, and it's bound to localhost ā which is the\npoint, so reach it over SSH rather than opening a port:
\nssh -L 4000:127.0.0.1:4000 ubuntu@your-wec-instance
Leave that running and open http://localhost:4000/ui. Username admin, password\nis the master key from your .env.
![\"The](\"https://wec.wiline.com/docs/assets/images/gw-ui-login-94d370f9590bd022684fb80ccabb2aff.png\")
\nFigure 6. The login page states its own default credentials, which is a good\nreminder that the master key is the only thing standing there.
Click Models + Endpoints:
\n![\"Model](\"https://wec.wiline.com/docs/assets/images/gw-ui-models-4372c39b1bfb45a2d61782e145f1c5e2.png\")
\nFigure 7. All four aliases registered, each mapped to its openai/... target.
The master key can reach every model and create more keys. No application should\never hold it. Instead, issue a virtual key scoped to what that app actually needs.
\nVirtual Keys ā + Create New Key:
\n| Field | Value |
|---|---|
| Owned By | You |
| Team | leave empty |
| Key Name | demo-app |
| Models | qwen-small |
| Max Budget (USD) | 0.10 |
| Reset Budget | daily |
Two things about this form. Max Budget lives under Optional Settings, which\nstarts collapsed ā easy to miss and then wonder where the budget field went. And\nleaving Models empty means all models, the opposite of what you want here.
\n![\"The](\"https://wec.wiline.com/docs/assets/images/gw-key-create-f440427f7d6f55ad01482ed469ff3624.png\")
\nFigure 8. Scoped to one model, capped at ten cents a day.
![\"The](\"https://wec.wiline.com/docs/assets/images/gw-key-saved-e13c0abfcd4b4c0bfefa791764eaae50.png\")
\nFigure 9. The dialog says it plainly. Believe it.
The generated key is shown once and never again. Close this without copying and\nyour only option is to delete the key and create another.
Now the part that justifies the whole exercise. Same key, two models:
\ncurl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $VK\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-small\",\"messages\":[{\"role\":\"user\",\"content\":\"say hi\"}],\"max_tokens\":20}' | jq -r '.choices[0].message.content'
Hi there! How can I assist you today?
curl -s http://127.0.0.1:4000/v1/chat/completions \\
-H \"Authorization: Bearer $VK\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"qwen-large\",\"messages\":[{\"role\":\"user\",\"content\":\"say hi\"}],\"max_tokens\":20}' | jq
{
\"error\": {
\"message\": \"key not allowed to access model. This key can only access models=['qwen-small']. Tried to access qwen-large\",
\"type\": \"key_model_access_denied\",
\"param\": \"model\",
\"code\": \"403\"
}
}
![\"The](\"https://wec.wiline.com/docs/assets/images/gw-key-403-ff8ff7ee956b8b081dd67c4d3834ac10.png\")
\nFigure 10. One credential, one model. Leak it and the blast radius is that one line.
Compare that to the situation we started in: a key in five .env files, with access\nto all thirteen models and no way to tell which app is using it.
You can issue a scoped credential per application, restrict it to specific models\nwith a spend cap, and revoke it on its own without touching anything else.
Every call lands in a spend log:
\ncurl -s http://127.0.0.1:4000/spend/logs \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
| jq '[.[] | select(.spend > 0)] | .[-3:] | .[] | {model, spend, prompt_tokens, completion_tokens}'
{
\"model\": \"openai/Qwen3.5-9B\",
\"spend\": 8.520000000000001e-05,
\"prompt_tokens\": 16,
\"completion_tokens\": 20
}
{
\"model\": \"openai/Qwen3.5-9B\",
\"spend\": 2.62e-05,
\"prompt_tokens\": 11,
\"completion_tokens\": 5
}
![\"Spend](\"https://wec.wiline.com/docs/assets/images/gw-spend-logs-dfdc834679299ecd1fcfcc774e83b5b3.png\")
\nFigure 11. Tokens counted per call, with a price attached to each.
That select(.spend > 0) isn't cosmetic. Failed calls are logged too, at zero\ntokens and zero spend ā the 401s from earlier are all sitting in there with\nstatus: \"failure\". Useful on its own: the spend log doubles as a record of\nsomeone trying keys that don't work.
Token counts are real. The dollar figures are not ā not for you, anyway. These are\nself-hosted models on your own infrastructure; the gateway is pricing them off a\nbuilt-in table of public API rates. The number is precise and confidently wrong,\nwhich is worse than blank. Fixing that means declaring your own cost per token, and\nthat's the subject of the fourth post in this series.
\nTwo more things in the full log entry, both useful:
\n\"litellm_overhead_time_ms\": 36.189,
\"messages\": {},
\"response\": {}
The gateway measures its own overhead and reports it per request ā 36ms here, which\nis a real number you can hold it to. And prompts and responses aren't stored by\ndefault: the log knows a call happened and what it cost, not what was said.
\nAlmost always the database. Check docker compose logs litellm for migration\nerrors ā if the gateway came up before Postgres was ready, the depends_on\ncondition is missing or the healthcheck isn't passing.
content comes back null with finish_reason: \"length\"āThe model is a reasoning model and it used your whole token budget thinking. Raising\nmax_tokens helps but doesn't guarantee anything ā the same prompt at 200 tokens\nanswered on one run and returned null on the next. For work that doesn't need\nreasoning, send it to a model that doesn't do any.
It's bound to 127.0.0.1, so it's unreachable from anywhere but the box itself.\nUse an SSH tunnel. If the tunnel is up and the page still won't load, confirm the\nport in docker compose ps matches the one you forwarded.
Chains joined with && stop at the first command that fails, and a failed grep\ncounts as a failure. Sourcing an env file whose variable name you guessed wrong\nleaves a placeholder in place and everything looks fine until an auth error three\nsteps later. Check for placeholders explicitly:
grep -c 'your-wec-key-here' .env
From the terminal, by alias:
\ncurl -s -X POST http://127.0.0.1:4000/key/delete \\
-H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\
-H \"Content-Type: application/json\" \\
-d '{\"key_aliases\":[\"demo-app\"]}' | jq
You now have one endpoint, a credential per app, and a log of every call. The\nobvious question is the one Figure 4 raised: if the small model answers in 3 tokens\nwhat the reasoning model spends 200 on without answering at all, why is anything\nrouting to the expensive one by default?
\nThe next post puts the gateway in charge of that decision, and measures what asking\nit costs.
\nPart 1 of this series audited a live\nmulti-agent box and found every network-exposure question worth asking. One line from\nthat audit didn't get followed up: \"the Postgres, ClickHouse and Redis behind Langfuse\nwere published to 127.0.0.1 instead of the world ā someone made a good decision\nthere. Hold that thought.\"
Here's the other half of that thought: getting the network right says nothing about\nwhat happens after someone's already inside a container. If the process running in\nthere is root, a compromise starts with the keys to the whole filesystem. So we checked\nā on the same box, the same Langfuse stack Part 1 already praised ā whether \"network\ncorrect\" also meant \"privilege correct.\" It didn't, for two of the six containers. Here's\nwhat fixing that actually looked like, including the part that broke.
\n\nDocker doesn't tell you this by default ā a container can be published correctly,\nhealthy, and running as root the entire time, with nothing in docker ps hinting at\nit. The only way to know is to ask the container directly:
docker exec langfuse-postgres-1 id -u
docker exec langfuse-redis-1 id -u
0
0
![\"Both](\"https://wec.wiline.com/docs/assets/images/privsec-root-confirmed-9af63f21bf8c07c7554a6cbab4f02929.png\")
\nFigure 1. The check that most setups never run: both id -u calls come back 0.\nHealthy, published correctly, and root the entire time.
0 is root. Both were running as root, and looking at the compose file explains why ā\nneither service has a user: directive at all:
redis:
image: docker.io/redis:7
restart: always
command: >
--requirepass ${REDIS_AUTH:-myredissecret}
--maxmemory-policy noeviction
...
postgres:
image: docker.io/postgres:${POSTGRES_VERSION:-17}
restart: always
...
Nobody decided these should run as root. It's just what happens when you don't set\nanything ā same shape as Part 1's finding that an unset Redis password defaults to no\nauthentication at all. The clickhouse service in the same file shows the fix already\nhalf-done, sitting right there for comparison:
clickhouse:
image: docker.io/clickhouse/clickhouse-server
restart: always
user: \"101:101\"
One line. Somebody set it for ClickHouse and never got to the other two.
\nDon't invent a UID ā both official images already ship a non-root user built for exactly\nthis, and guessing wrong means either a permission error or, worse, a UID that silently\ndoesn't match the one the image's own files are owned by:
\ndocker exec langfuse-postgres-1 id postgres
docker exec langfuse-redis-1 id redis
uid=999(postgres) gid=999(postgres) groups=999(postgres),101(ssl-cert)
uid=999(redis) gid=999(redis) groups=999(redis)
Both images happen to use 999:999 for their built-in non-root user. Add it to the\ncompose file:
redis:
image: docker.io/redis:7
user: \"999:999\"
restart: always
...
postgres:
image: docker.io/postgres:${POSTGRES_VERSION:-17}
user: \"999:999\"
restart: always
...
docker compose up -d --force-recreate postgres redis
ā Container langfuse-postgres-1 Recreated
ā Container langfuse-redis-1 Recreated
sleep 5
docker compose ps
docker exec langfuse-postgres-1 id -u
docker exec langfuse-redis-1 id -u
langfuse-postgres-1 Up 34 seconds (healthy) 127.0.0.1:5433->5432/tcp
langfuse-redis-1 Up 34 seconds (healthy) 127.0.0.1:6379->6379/tcp
999
999
![\"docker](\"https://wec.wiline.com/docs/assets/images/privsec-nonroot-healthy-2aa6142e51931837aa80ee308e8be448.png\")
\nFigure 2. Same two containers, seconds after --force-recreate: both healthy, both\nid -u now 999 instead of 0.
No permission errors, no ownership drama, both healthy within seconds. That's worth\nsaying plainly because it's not always true ā if these volumes had been initialized as\nroot somewhere upstream and never chown-ed, this exact command can fail with\npermission denied on the data directory. It didn't here, but check your own logs after\nrecreating, don't assume \"no output\" means \"no problem.\"
You can check whether a running container is actually root (docker exec <container> id -u), find its image's intended non-root user instead of guessing a UID\n(docker exec <container> id <username>), and convert it with a one-line user: in\ncompose.
Everything above was clean. Then the logs from a completely different container told a\ndifferent story:
\ndocker logs langfuse-langfuse-worker-1 --tail 20
Error: getaddrinfo ENOTFOUND redis
at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:122:26)
Redis error getaddrinfo ENOTFOUND redis
Redis error connect ECONNREFUSED 172.19.0.5:6379
Queue job mixpanel-integration-processing-queue errored: Error: Socket timeout.
Expecting data, but didn't receive any in 30000ms.
Queue job trace-delete errored: Error: Socket timeout. Expecting data, but didn't
receive any in 30000ms.
![\"langfuse-worker's](\"https://wec.wiline.com/docs/assets/images/privsec-worker-broken-e776f29e677755bd8a5e61864dbe22bc.png\")
\nFigure 3. A container nobody touched, failing anyway ā every queue job erroring with\nthe same 30-second socket timeout against Redis, well after the recreate that caused it.
langfuse-worker never touched Postgres or Redis's ownership ā it just holds a live\nconnection to Redis, and --force-recreate tore down the container that connection\npointed at. Docker's own healthcheck said Redis was healthy again well before the\nworker gave up retrying; ioredis just doesn't reconnect cleanly on its own here, and\nkept failing with 30-second socket timeouts long after the dependency it depended on was\nback.
The fix is the dependent service's own restart, not another look at Redis:
\ndocker compose restart langfuse-worker langfuse-web
2026-08-10T19:40:22.505Z info Redis connection has been closed.
2026-08-10T19:40:22.769Z info Shutdown complete, exiting process...
sleep 10
docker logs langfuse-langfuse-worker-1 --tail 15
experiment-create-queue executor started: true
posthog-integration-queue executor started: true
data-retention-queue executor started: true
webhook-queue executor started: true
Listening: http://21caf9a28775:3030
![\"langfuse-worker's](\"https://wec.wiline.com/docs/assets/images/privsec-worker-recovered-ca07f68b5e4cb0994524f20c497f41ca.png\")
\nFigure 4. Same container, one restart later ā every queue executor starts clean, no\nRedis errors anywhere in the log.
Clean queue startup, no Redis errors. That's the whole lesson: hardening a container\ndoesn't stay contained to that container. Anything holding a live connection to the\nthing you just recreated needs its own restart, on purpose, not as an afterthought you\ndiscover from an error log twenty minutes later.
\n--force-recreate on a shared dependency (a database, a queue, a cache) can silently\nbreak every service that already had a connection open to it ā even after the\ndependency itself reports healthy again. Restart dependents explicitly; don't assume\nthey'll notice on their own.
| Container | Before | Fix | Broke anything? |
|---|---|---|---|
langfuse-postgres-1 | root (uid 0) | user: \"999:999\" | No ā recreated clean |
langfuse-redis-1 | root (uid 0) | user: \"999:999\" | No ā recreated clean |
langfuse-clickhouse-1 | already 101:101 | none needed | ā |
langfuse-langfuse-worker-1 | untouched | none ā just needed a restart | Yes ā lost its Redis connection after the other two containers recreated |
Two one-line fixes. The actual cost wasn't the fix ā it was noticing the third\ncontainer that nobody touched was the one that broke.
\ndocker compose restart said \"no configuration file provided: not found\"āYou're not in the project directory Compose expects ā it only finds docker-compose.yml\nrelative to your current working directory (or via -f). cd into the stack's own\nfolder first; this exact error means Compose never even looked at your containers.
Restarting the dependency doesn't help ā the connection that broke belongs to the\ndependent service. Restart the thing holding the connection (here, langfuse-worker),\nnot the thing it connects to.
docker exec <container> id <username> returns \"no such user\"āThe image doesn't ship a dedicated non-root user under that name ā check the image's own\ndocumentation, or docker exec <container> cat /etc/passwd to see what's actually\navailable before picking a UID.
Privilege is the second layer: network hardening keeps a compromise from spreading\nbetween stacks, non-root keeps a compromise from owning the whole container. The last\nlayer in this series is identity ā putting real authentication in front of the ports\nthat must stay exposed, which is where NetBird\nand Caddy come in.
\nYou start with one agent. Then it needs a database. Then you add a second agent, a\nmessaging bridge, an observability stack. Six months later a single WEC Instance is running\nfive compose projects, twenty-something containers, and nobody remembers which ports are\nopen to the world.
\nThat's not a hypothetical ā that's the box this tutorial was written on. So instead of\ntheorizing, we probed it: can containers reach each other across stacks? Can they reach the\ndatabases? Is the firewall actually protecting anything?
\nThree of the answers surprised me. One of them was a database sitting there with no\npassword. And the firewall ā the firewall was lying.
\n\nFive independent compose projects, deployed over months, each with its own network (host, none and bridge are Docker's built-ins, plus one leftover from a service that isn't running):
docker network ls
NETWORK ID NAME DRIVER SCOPE
655850b24125 bridge bridge local
05e5d49e7e9d evolution-api_default bridge local
e8bd75ecbb0a host host local
ce697baac173 langfuse_default bridge local
ead3e0822ce6 nlp-evaluation-service_nlp-net bridge local
5d0ab32d14d7 none null local
9a6727150f2b openclaw_default bridge local
ec77b98f457b rag-service_default bridge local
5f88cabf97f0 remark42_default bridge local
And here's what's published to the world:
\ndocker ps --format '{{.Names}}\\t{{.Ports}}' | grep '0.0.0.0'
remark42-remark42-1 0.0.0.0:8082->8080/tcp, [::]:8082->8080/tcp
evolution-api-evolution-api-1 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp
langfuse-langfuse-web-1 0.0.0.0:3001->3000/tcp, [::]:3001->3000/tcp
langfuse-minio-1 0.0.0.0:9090->9000/tcp, [::]:9090->9000/tcp, 127.0.0.1:9091->9001/tcp
![\"The](\"https://wec.wiline.com/docs/assets/images/netsec-exposure-audit-904ef03d2c7552d9b1b8c9d9e69218dd.png\")
\nFigure 1. The start of any audit. Nine networks: three Docker built-ins (bridge, host,\nnone), five compose projects, and one leftover. Four services published on 0.0.0.0 ā every\ninterface, including the public one.
Four services listening on every interface. Note what's not in that list: the Postgres,\nClickHouse and Redis behind Langfuse. Those were published to 127.0.0.1 instead ā someone\nmade a good decision there. Hold that thought.
The classic worry: OpenClaw gets compromised, and the attacker walks into Langfuse's\ndatabase next door. Let's find out instead of guessing. Grab the container IPs:
\nfor c in langfuse-postgres-1 evolution-api-evolution-postgres-1; do
docker inspect -f '{{.Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $c
done
/langfuse-postgres-1 172.19.0.5
/evolution-api-evolution-postgres-1 172.23.0.4
Now try to reach both from inside the OpenClaw container, by IP ā no DNS, no shortcuts:
\ndocker exec openclaw-openclaw-gateway-1 node -e '
const net=require(\"net\");
[[\"langfuse-postgres\",\"172.19.0.5\",5432],[\"evolution-postgres\",\"172.23.0.4\",5432]]
.forEach(([n,ip,p])=>{
const s=net.connect({host:ip,port:p,timeout:4000});
s.on(\"connect\",()=>{console.log(`REACHED ${n}`);s.destroy()});
s.on(\"timeout\",()=>{console.log(`blocked ${n}`);s.destroy()});
s.on(\"error\",e=>console.log(`error ${n} ${e.code}`));
});'
blocked langfuse-postgres
blocked evolution-postgres
![\"Cross-network](\"https://wec.wiline.com/docs/assets/images/netsec-isolation-blocked-52d0644c0dbd63427ca62b09eb5b9947.png\")
\nFigure 2. Connecting by raw IP from one stack's container to another stack's database:\nboth time out. Docker's isolation rules are doing their job.
Blocked. Modern Docker installs DOCKER-ISOLATION-STAGE rules that drop traffic between\nuser-defined bridge networks, and they work. If you've read that \"Docker networks don't\nreally isolate,\" test it on your own box before believing it ā on current Docker they do.
You can verify network isolation empirically instead of trusting folklore: get the target's\nIP with docker inspect, then connect by IP from inside another container.
So cross-stack isolation is fine. Which means the real risk is somewhere else.
\nEach compose project puts all its services on one network. That's the default, and it means\nthe application container can reach every backing service ā which is intended. The\nquestion is what that access actually looks like. From Evolution API's bridge container:
\ndocker exec evolution-api-bridge-1 python3 -c '
import socket
for name, host, port in [(\"postgres\",\"evolution-postgres\",5432),
(\"redis\",\"evolution-redis\",6379),
(\"api\",\"evolution-api\",8080)]:
s=socket.socket(); s.settimeout(4)
try:
s.connect((host,port)); print(f\"REACHED {name}:{port}\")
except Exception as e:
print(f\"blocked {name}:{port} -- {type(e).__name__}\")
finally: s.close()'
REACHED postgres:5432
REACHED redis:6379
REACHED api:8080
Reachable is expected. Unauthenticated is not. Check whether those services actually ask\nfor credentials ā first Redis:
\ndocker exec evolution-api-bridge-1 python3 -c '
import socket
s=socket.socket(); s.settimeout(5); s.connect((\"evolution-redis\",6379))
s.sendall(b\"INFO server\\r\\n\")
print(s.recv(200).decode(errors=\"replace\")[:120])'
$625
# Server
redis_version:7.4.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b61b4eb609520881
redis_
![\"Redis](\"https://wec.wiline.com/docs/assets/images/netsec-redis-noauth-22e25e2b5bbf007d514a914ec9ec8b91.png\")
\nFigure 3. The finding: INFO server answered with no credentials (output truncated at 120\nchars by the probe). Any container on this network can read or wipe the session store.
That's a INFO command answered with no AUTH ā the Redis holding this deployment's\nsession state has no password. Anything that gets code execution in any container on that\nnetwork can read it, write to it, or FLUSHALL it.
Postgres, in the same stack, behaves correctly:
\ndocker exec evolution-api-bridge-1 python3 -c '
import socket,struct
s=socket.socket(); s.settimeout(5); s.connect((\"evolution-postgres\",5432))
msg=b\"user\\x00postgres\\x00database\\x00postgres\\x00\\x00\"
s.sendall(struct.pack(\"!ii\",len(msg)+8,196608)+msg)
print(s.recv(64))'
b'R\\x00\\x00\\x00\\x17\\x00\\x00\\x00\\nSCRAM-SHA-256\\x00\\x00'
It demands SCRAM-SHA-256. Same box, same network, two backing services ā one asks for a\npassword, the other doesn't. Nobody decided that; it's just what the images default to when\nyou don't set a password.
\nThe lesson: \"internal network\" is not a security boundary. It's a convenience. Treat\nevery service as if the attacker is already on that network, because if one container falls,\nthey are.
\nNow the big one. Turn on UFW and deny everything inbound except SSH:
\nsudo ufw allow 22/tcp # ALWAYS first, or you lock yourself out
sudo ufw default deny incoming
sudo ufw --force enable
sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip
To Action From
-- ------ ----
60000:61000/udp ALLOW IN Anywhere
22/tcp ALLOW IN Anywhere
60000:61000/udp (v6) ALLOW IN Anywhere (v6)
22/tcp (v6) ALLOW IN Anywhere (v6)
Deny incoming ā only SSH and mosh's UDP range allowed. Now, from a different machine,\nopen the Langfuse UI and the Evolution API:
\nhttp://<instance-ip>:3001 -> Langfuse UI loads normally
http://<instance-ip>:8080 -> {\"status\":200,\"message\":\"Welcome to the Evolution API...\"}
![\"The](\"https://wec.wiline.com/docs/assets/images/netsec-ufw-bypass-130569873d3680226c3534f016c4756a.png\")
Figure 4. The firewall says it denies everything inbound. The Evolution API on :8080\nanswers anyway ā status 200, straight from a browser on another machine.
Both wide open, through a firewall configured to block them. This is the single most common\nsecurity surprise in self-hosted Docker, and it isn't a bug.
\nLook at the order of the FORWARD chain:
sudo iptables -L FORWARD -n --line-numbers | head -8
Chain FORWARD (policy DROP)
num target prot opt source destination
1 DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
2 DOCKER-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
5 ufw-before-logging-forward all -- 0.0.0.0/0 0.0.0.0/0
6 ufw-before-forward all -- 0.0.0.0/0 0.0.0.0/0
![\"UFW](\"https://wec.wiline.com/docs/assets/images/netsec-ufw-chains-7a5553d8ce1882eb415c7e12e1beb065.png\")
\nFigure 5. The whole explanation in one screen: the firewall says deny incoming, and\nDocker's chains sit at positions 1-2 while UFW's start at 5.
Docker's chains sit at positions 1 and 2. UFW's don't appear until 5 and 6. And\npublishing a port isn't a listener on the host ā it's a NAT rule:
\nsudo iptables -t nat -L DOCKER -n | grep -E '3001|8080'
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 to:172.23.0.3:8080
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001 to:172.19.0.2:3000
The packet arrives, Docker's NAT rewrites the destination to the container, and it becomes\nforwarded traffic that Docker's own chain accepts ā long before any UFW rule is consulted.\nYour UFW rules aren't ignored; they're simply never reached.
\nufw deny does not protect a published Docker port. If you've been relying on UFW in\nfront of -p 8080:8080, that port has been open the whole time.
The cleanest fix isn't a firewall rule; it's not opening the door. If a service is only\nconsumed by other containers, it needs no ports: entry ā service-to-service traffic\nworks over the compose network by name.
If you need it reachable from the host (a debugging port, a local psql), bind it to\nloopback explicitly:
\nservices:
postgres:
# 127.0.0.1 prefix = host-only, never the public interface
ports:
- \"127.0.0.1:5433:5432\"
That's exactly what the Langfuse stack on this box already does ā its Postgres, ClickHouse\nand Redis are all on 127.0.0.1, which is why they never showed up in our exposure audit.\nThe default \"5433:5432\" means 0.0.0.0 ā every interface, including the public one.
internalāOne network per compose project is the default, not a design. Give the backing services\ntheir own network, and put only the application in both:
\nservices:
app: # talks to the world AND to the database
image: your-agent
networks: [frontend, backend]
db:
image: redis:7
command: redis-server --requirepass ${REDIS_PASSWORD}
networks: [backend] # backend only ā no route out
public: # e.g. a webhook receiver
image: your-public-thing
networks: [frontend] # cannot see the database at all
networks:
frontend:
backend:
internal: true # no gateway: no inbound, no outbound
Verified on the same box:
\n# the frontend-only container tries to reach the database
docker exec netsecdemo-public-1 python3 -c '
import socket
s=socket.socket(); s.settimeout(4)
try: s.connect((\"db\",6379)); print(\"REACHED db <- leak\")
except Exception as e: print(f\"blocked: {type(e).__name__}\")'
# the app, which is on both networks
docker exec netsecdemo-app-1 python3 -c '
import socket
s=socket.socket(); s.settimeout(4)
try: s.connect((\"db\",6379)); print(\"REACHED db <- correct, it needs it\")
except Exception as e: print(f\"blocked: {type(e).__name__}\")'
# and can the database itself reach the internet?
docker exec netsecdemo-db-1 timeout 6 getent hosts pypi.org >/dev/null \\
&& echo \"resolved external DNS\" || echo \"no internet egress <- internal working\"
blocked: gaierror
REACHED db <- correct, it needs it
no internet egress <- internal working
![\"The](\"https://wec.wiline.com/docs/assets/images/netsec-internal-demo-6231d6936686cf9d282c7f494797a185.png\")
Figure 6. Three properties proven in one run ā the public-facing container can't even\nresolve the database, the app can reach it, and the database has no route to the internet.
\nThree properties at once: the public-facing container can't even resolve the database, the\napp can, and the database can't phone home ā which matters the day a dependency of yours\nturns malicious.
\nOur audit found Redis answering INFO with no credentials. One line fixes it:
db:
image: redis:7
command: redis-server --requirepass ${REDIS_PASSWORD}
-NOAUTH Authentication required.
That's the real output from this box, after applying the fix ā the finding from Test 2 is\nclosed. If you set a password on a Redis that something already uses, remember to update the\nclient's connection string too (redis://:<password>@host:6379/0), or you'll trade an open\ndatabase for a broken one.
Do this even for services that aren't published. Defense in depth means the second layer\nholds when the first one fails.
\nDOCKER-USERāSometimes a port genuinely has to be public-facing but restricted by source. The chain to\nuse is DOCKER-USER ā Docker guarantees it runs before its own accept rules, and never\noverwrites it:
# only 203.0.113.0/24 may open new connections to the container's port 3000
sudo iptables -I DOCKER-USER 1 -p tcp --dport 3000 \\
-m conntrack --ctstate NEW ! -s 203.0.113.0/24 -j DROP
Tested live on the Langfuse port, from a browser on another machine:
\nhttp://<instance-ip>:3001 -> Langfuse UI loads
ERR_CONNECTION_TIMED_OUT
![\"The](\"https://wec.wiline.com/docs/assets/images/netsec-docker-user-blocked-8e34c54fd3b4552806e49bf87aed7cfb.png\")
Figure 7. Same URL, same firewall, one rule in the right chain ā and the port is finally\nclosed. Note Chrome's own suggestion: \"Checking the proxy and the firewall.\" This time the\nfirewall really is the answer.
\nMeanwhile the Evolution API on :8080, which we deliberately left alone, kept answering.\nThe rule is surgical, and it works exactly where UFW couldn't.
iptables -I is not persistent. Use iptables-persistent (netfilter-persistent save), or\nput the rule in a small systemd unit that runs after docker.service. A firewall rule that\ndisappears on reboot is worse than none, because you'll think you're protected.
You can audit and lock down a multi-stack Docker host: prove what's reachable, bind or split\nwhat shouldn't be, authenticate the backends, and filter published ports in the one chain\nDocker respects.
Run these four commands on any box you own. They take a minute and tell you more than any\narchitecture diagram:
\n# 1. What's exposed to the world?
docker ps --format '{{.Names}}\\t{{.Ports}}' | grep '0.0.0.0'
# 2. Are your firewall rules even in the path? (Docker chains before ufw = they aren't)
sudo iptables -L FORWARD -n --line-numbers | head -8
# 3. What can a compromised container reach on its own network?
docker exec <container> python3 -c \"import socket;s=socket.socket();s.settimeout(3);s.connect(('<svc>',<port>));print('reached')\"
# 4. Does that backing service ask for a password?
docker exec <container> python3 -c \"
import socket;s=socket.socket();s.settimeout(4);s.connect(('redis',6379))
s.sendall(b'INFO server\\r\\n');print(s.recv(60))\"
| Finding | Severity | Fix |
|---|---|---|
| Cross-network isolation works | ā none | nothing ā Docker already does this |
| Every service on one network, mutually reachable | ā ļø medium | split by trust zone, internal: true |
Redis answering with no AUTH | š“ high | one line: --requirepass |
ufw deny not applied to published ports | š“ high | bind to 127.0.0.1, or DOCKER-USER rule |
Backing DBs on 127.0.0.1 (Langfuse) | ā none | already correct ā copy this pattern |
None of the fixes took more than a line of YAML. The expensive part was knowing which line\nā and the only way to know is to probe your own box instead of trusting the diagram in your\nhead.
\nDOCKER-USER rule and nothing changedāTwo usual causes. First, --dport must be the container's port, not the published one ā\nthe DNAT already rewrote the destination by the time DOCKER-USER sees the packet (in our\ncase 3000, not 3001). Second, existing connections aren't affected: the\n--ctstate NEW match only touches new ones, so an already-open browser tab keeps working.
sudo ufw allow 22/tcp before ufw enable, always. If you're already locked out, most\nproviders give you a serial/VNC console ā use it to run ufw disable.
internal: true broke my container's package installāThat's it working. An internal network has no gateway, so no egress at all ā no pip install, no DNS, no calling an external API. Backing services shouldn't need any of that;\nif a container does, it belongs on the frontend network too.
Compose DNS only resolves names within a shared network. If app and db are on\ndifferent networks with nothing in common, db won't resolve. The app has to be a member of\nboth, as in Fix 2.
Network hardening is the layer that keeps a compromised container from becoming a\ncompromised host. The next layers, in the order I'd do them: run containers as non-root\n(user: in compose) so a breakout starts with fewer privileges, and put an identity\nprovider in front of the ports you must expose ā a private mesh with\nNetBird plus authentication is the natural pairing\nwith everything above.
![\"Hermes](\"https://wec.wiline.com/docs/assets/images/hermesagent-c76bbe81610ae2e70b0950e0f96bec8a.png\")
+In Part 1 you self-hosted Hermes with\npersistent memory. This part connects it to the same Telegram bot from the\nOpenClaw series ā the chat your\nusers already know keeps working exactly as it did, just a different agent\nanswering underneath. No new bot to announce, no channel to migrate people to.
\n\nContinues on the same WEC Instance as Part 1. Hermes gateway is already\ninstalled and running as a systemd user service ā this part is configuration,\nnot installation. Every command and error below is from the actual run.
Self-hosting is only worth the effort if it doesn't lock you into whatever\nagent you started with. The bot itself ā its name, its chat history, the\npeople who already talk to it ā is the asset. The agent behind it is\nreplaceable. Swapping OpenClaw for Hermes on the same bot proves that: same\nTelegram front door, same conversation thread, new brain.
\nhermes gateway setup
Pick Telegram from the platform list, then choose how to connect it:
\nā Telegram
How would you like to create your Telegram bot?
[1] Automatic (recommended)
Scan a QR code ā confirm in Telegram ā done.
No token copy-paste needed.
[2] Manual
Create a bot via @BotFather yourself and paste the token.
Choice [1/2] [1]: 2
Create a bot via @BotFather on Telegram
Telegram bot token: **********************************************
ā Telegram token saved
Take option 2 ā option 1 spins up a brand new bot via QR code, which\ndefeats the point if you're reusing an existing one.
\nNext the wizard asks who's allowed to use the bot:
\nš Security: Restrict who can use your bot
To find your Telegram user ID:
1. Message @userinfobot on Telegram
2. It will reply with your numeric ID (e.g., 123456789)
Allowed user IDs (comma-separated, leave empty for open access):
If you don't have your ID handy, leave it blank and move on ā you can lock\nthis down later. But the prompt is misleading: leaving it empty doesn't\nopen access by default. The gateway logs told the real story after restart:
\nWARNING gateway.run: No user allowlists configured. All unauthorized users
will be denied. Set GATEWAY_ALLOW_ALL_USERS=true in ~/.hermes/.env to allow
open access, or configure platform allowlists (e.g.,
TELEGRAM_ALLOWED_USERS=your_id).
An empty allowlist means deny everyone, not open access. If you want open\naccess (fine for a personal bot, not for anything public-facing), you need\none more step:
\necho 'GATEWAY_ALLOW_ALL_USERS=true' >> ~/.hermes/.env
The wizard also asks for a Home Channel ā where Hermes delivers cron\nresults and cross-platform messages. Leave it blank too; you can set it later\nfrom inside the chat with /sethome.
Finish the wizard and let it restart the gateway:
\nRestart the gateway to pick up changes? [Y/n]: Y
ā User service restarted
![\"Terminal](\"https://wec.wiline.com/docs/assets/images/hermes-gateway-setup-wizard-7293ce9021ee147b93bad2dcb7a72a53.png\")
\nFigure 1. The whole wizard, start to finish ā token, the allowlist warning, and the restart.
If you added GATEWAY_ALLOW_ALL_USERS after the wizard already restarted\nonce (as above), restart again so it picks up the .env change:
systemctl --user restart hermes-gateway.service
Message your bot. Not @BotFather ā your bot, by the username you gave it\nwhen you created it. It's an easy mix-up: BotFather answers everything with\nits own fixed command menu, so if you see that instead of a real reply,\nyou're in the wrong chat.
\nThe first message may get a system nudge instead of a chat reply ā that's\nHermes noticing there's no home channel yet, not a failure:
\nš¬ No home channel is set for Telegram. A home channel is where Hermes
delivers cron job results and cross-platform messages.
Type /sethome to make this chat your home channel, or ignore to skip.
Send another message and it answers for real:
\n![\"Telegram](\"https://wec.wiline.com/docs/assets/images/hermes-telegram-verified-17c22677d5e1e6a1d6147099b58fdc53.png\")
\nFigure 2. The same bot from the OpenClaw series, now answering through Hermes.
Ask it something with actual depth, and it delivers a real, structured answer\nā not just a canned reply:
\n![\"Telegram](\"https://wec.wiline.com/docs/assets/images/hermes-telegram-sky-blue-139494c527c4075440bf7afa25c14039.png\")
\nFigure 3. Same agent, a real question ā full explanation, not a one-liner.
Same bot, same chat history, new agent.
\n\"No user allowlists configured. All unauthorized users will be denied.\" ā\nleaving the allowlist prompt empty does not mean open access; it means\ndeny-all. Set GATEWAY_ALLOW_ALL_USERS=true in ~/.hermes/.env and restart\nthe gateway.
\"Telegram polling conflict... make sure that only one bot instance is\nrunning\" ā shows up for a few seconds right after a restart, while\nTelegram's servers finish releasing the previous session. It retries and\nclears on its own within 20 seconds; no action needed unless it keeps\nrepeating past the 5th retry, which means another process really is polling\nthe same token.
\nBot doesn't respond at all ā check you're messaging the bot itself, not\n@BotFather. BotFather always answers with its own command list regardless of\nwhat you send it.
\nThe bot answers ā next is putting it on a schedule: a morning briefing or a\nservice-health check delivered to this same chat, no polling required.
\n![\"SearXNG\"](\"https://wec.wiline.com/docs/assets/images/searxng-mark-1b536bb949960590cea598d56becda08.png\")
+Say you've built a support chatbot or a RAG assistant on WEC. It's solid on your docs and\non what the model already knows ā but ask it something current (\"what's the latest\nrelease of X?\", today's pricing, a recent change) and it either guesses or tells you its\nknowledge is out of date. For anything that needs to stay current, that's a real gap.
\nWEC Inference now has a web-search tool built in. Add it to a normal\n/v1/chat/completions call and the model can look things up on the live web ā the search\nruns on WiLine's own infrastructure (SearXNG), so nothing goes out to a third-party search\nvendor. Let's test it.
Calls hit https://inference.wiline.com/v1/chat/completions with your WEC Inference API\nkey, model Qwen3.5-9B. Nothing to install ā the tool is part of the API.
A normal call, asking something recent:
\ncurl -sS https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{
\"model\": \"Qwen3.5-9B\",
\"messages\": [{\"role\": \"user\", \"content\": \"What is the latest stable version of Docker Engine, and when was it released?\"}]
}'
![\"The](\"https://wec.wiline.com/docs/assets/images/websearch-without-aeaee1a5d9a326ccb92d56a8ec08bdc5.png\")
\nFigure 1. With no tool, the model won't commit to a version ā it says its knowledge is\nlimited to its training cutoff and points you to the release notes instead.
The same request, plus a web_search tool and tool_choice: auto:
curl -sS https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{
\"model\": \"Qwen3.5-9B\",
\"messages\": [{\"role\": \"user\", \"content\": \"What is the latest stable version of Docker Engine, and when was it released?\"}],
\"tools\": [{
\"type\": \"function\",
\"function\": {
\"name\": \"litellm_web_search\",
\"description\": \"Search the web for current information\",
\"parameters\": {
\"type\": \"object\",
\"properties\": {\"query\": {\"type\": \"string\", \"description\": \"The search query\"}},
\"required\": [\"query\"]
}
}
}],
\"tool_choice\": \"auto\"
}'
![\"The](\"https://wec.wiline.com/docs/assets/images/websearch-with-ae28b4b9dbd33effc4676a36ce37a4e0.png\")
\nFigure 2. Same model, same question, plus the tool ā now it returns a concrete version\nand release date pulled from the live web. The model decided to search, WEC ran the query,\nand fed the results back before answering.
That's the whole feature: one tool in the request body, current answers out.
\nWeb search injects the results into your prompt, so a searched call costs more tokens than\na bare one (in our test, prompt tokens went from ~26 to ~2,776). Keep tool_choice: auto\nso the model only searches when it actually needs to ā it'll skip search for questions it\ncan already answer.
auto handles that for you.Point it at a RAG assistant or a chatbot and it can stay current without you wiring up a\nseparate search service.
\nAn agent that calls tools does two very different things: it reasons (\"I should\nsearch the docs\") and it acts (actually calls the tool). When something goes wrong,\nthe first question is always which layer failed ā did it think wrong, or did the doing\nbreak? A flat log can't answer that. A trace can.
\nIn this tutorial you build a small tool-calling agent on WEC Inference, instrument it\nwith Langfuse so every reasoning step and every tool call becomes an inspectable node,\nthen debug two real failures from the trace tree ā including the worst kind: a\nconfident, wrong answer that never throws an error. Every command, error, and screenshot\nbelow is from a real run.
\n\nBuilt on a WEC Instance with Docker, against WEC Inference (Qwen3.5-122B) and a\nself-hosted Langfuse from the\nobservability tutorial. The agent's tools\ninclude the RAG service from the\nRAG tutorial ā so this piece ties the\nwhole series together. All model calls stay on WEC; nothing leaves for a third-party API.
The dotted lines are the point: every step reports itself to Langfuse, so the agent's\nreasoning and its actions land as separate, inspectable nodes.
\nPrerequisites: a WEC Instance with Docker, a WEC Inference API key, a running\nLangfuse (public + secret key, host URL), and a tool the agent can call ā here the RAG\n/ask service. All commands run in ~/agent-tracing.
Rule: never instrument before you've seen the raw behavior. So v1 does the minimum ā\nask the model a question with a tool definition, and print what comes back. Set up the\nworkspace and the WEC key:
\nmkdir -p ~/agent-tracing && cd ~/agent-tracing
grep '^WEC_API_KEY=' ~/evolution-api/.env > .env
agent.py (v1):
import os, requests
WEC_URL = \"https://inference.wiline.com/v1/chat/completions\"
WEC_API_KEY = os.environ[\"WEC_API_KEY\"]
MODEL = \"Qwen3.5-122B\"
TOOLS = [{
\"type\": \"function\",
\"function\": {
\"name\": \"search_docs\",
\"description\": \"Search the WEC documentation for an answer to a question.\",
\"parameters\": {
\"type\": \"object\",
\"properties\": {\"query\": {\"type\": \"string\"}},
\"required\": [\"query\"],
},
},
}]
def call_model(messages):
r = requests.post(WEC_URL,
headers={\"Authorization\": f\"Bearer {WEC_API_KEY}\"},
json={\"model\": MODEL, \"messages\": messages, \"tools\": TOOLS, \"tool_choice\": \"auto\"},
timeout=120)
r.raise_for_status()
return r.json()[\"choices\"][0][\"message\"]
if __name__ == \"__main__\":
msg = call_model([{\"role\": \"user\", \"content\": \"How do I create a compute instance on WEC?\"}])
print(\"=== REASONING LAYER (what it thought) ===\")
print(msg.get(\"reasoning_content\") or \"(none)\")
print(\"\\n=== ACTION LAYER (what it decided to do) ===\")
for tc in msg.get(\"tool_calls\") or []:
print(f\"tool: {tc['function']['name']} args: {tc['function']['arguments']}\")
Run it in a container (same pattern as the rest of the series):
\nFROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir requests
COPY agent.py .
CMD [\"python\", \"agent.py\"]
services:
agent:
build: .
env_file: .env
docker compose run --rm --build agent
![\"The](\"https://wec.wiline.com/docs/assets/images/agent-reasoning-action-56b6f9e7c5b8aef031cd06c04187e291.png\")
\nFigure 1. WEC Inference returns the two layers in one response: reasoning_content\n(the thinking) and tool_calls (the decision). We don't have to infer the split ā the API\nhands it to us.
Qwen3.5-122B returns finish_reason: tool_calls with a proper tool_calls array, plus\na reasoning_content field containing the model's chain of thought before it acts. That\nreasoning field is exactly what makes the reasoning-vs-action split visible later.
v1 decided to search but didn't. Now actually call the tool (the RAG /ask service),\nfeed the result back, and let the model write the final answer. Add the tool + the loop:
import json
RAG_URL = \"http://<your-vm-ip>:8000/ask\" # the RAG service = our one tool
def search_docs(query):
r = requests.post(RAG_URL, json={\"question\": query}, timeout=120)
r.raise_for_status()
return r.json()[\"answer\"]
def run(user_input):
messages = [{\"role\": \"user\", \"content\": user_input}]
while True:
msg = call_model(messages)
if not msg.get(\"tool_calls\"):
print(\"\\n=== FINAL ANSWER ===\\n\" + (msg.get(\"content\") or \"\"))
return msg.get(\"content\")
messages.append({\"role\": \"assistant\", \"content\": msg.get(\"content\") or \"\",
\"tool_calls\": msg[\"tool_calls\"]})
for tc in msg[\"tool_calls\"]:
args = json.loads(tc[\"function\"][\"arguments\"])
print(f\"[reasoning] {(msg.get('reasoning_content') or '').strip()[:140]}\")
print(f\"[action] {tc['function']['name']}({args})\")
result = search_docs(args[\"query\"])
print(f\"[tool result] {result[:140]}...\")
messages.append({\"role\": \"tool\", \"tool_call_id\": tc[\"id\"], \"content\": result})
docker compose run --rm --build agent
![\"The](\"https://wec.wiline.com/docs/assets/images/agent-loop-b09953660869437421c9222da3f9b135.png\")
\nFigure 2. The complete loop ā the model reasons, calls search_docs, gets the RAG\nresult, and writes a grounded answer. It works. But a working terminal tells you nothing\nabout whether the reasoning was sound ā that's what we fix next.
The terminal flattens everything into one stream. Langfuse turns each function into a\nnode so you can see the structure. Reuse the Langfuse keys from the observability\ntutorial (same project as your RAG traces):
\ngrep -E '^LANGFUSE_(PUBLIC_KEY|SECRET_KEY|HOST)=' ~/rag-service/.env >> .env
Add the SDK (pip install ... langfuse) and decorate three functions ā that's the whole\ninstrumentation. @observe wraps a function as a span; as_type=\"generation\" marks the\nLLM calls so Langfuse captures model, tokens, and cost:
from langfuse import observe, get_client
@observe(as_type=\"generation\")
def call_model(messages):
...
data = r.json()
get_client().update_current_generation(model=MODEL, usage_details=data.get(\"usage\"))
return data[\"choices\"][0][\"message\"]
@observe()
def search_docs(query):
...
@observe()
def run(user_input):
...
if __name__ == \"__main__\":
run(\"How do I create a compute instance on WEC?\")
get_client().flush() # short-lived script: push traces before exit
docker compose run --rm --build agent
Open Langfuse ā Tracing ā Traces. First thing you notice: there are two traces per\nquestion.
\n![\"The](\"https://wec.wiline.com/docs/assets/images/trace-list-3a0d985919db7d12160c4109ab85dede.png\")
\nFigure 3. Distributed tracing, for free: your agent's run trace and the RAG service's\nown ask trace are separate ā each service instruments itself. Seeing both is how you\nfollow one request across service boundaries.
Open the newest run:
![\"The](\"https://wec.wiline.com/docs/assets/images/trace-tree-green-c19c35d6b1ba42189355bdd0e79d96f8.png\")
\nFigure 4. The agent turn as a tree: run ā call_model (decide) ā search_docs\n(act) ā call_model (answer). Langfuse also captured latency, token usage\n(801ā306), and a correctness score ā all for three decorators.
Click the first call_model and open its output:
![\"The](\"https://wec.wiline.com/docs/assets/images/trace-reasoning-content-c5e9876f6e9a640d45e9f5f54c5594cc.png\")
\nFigure 5. The reasoning layer, captured: reasoning_content (\"I should search the WEC\ndocumentationā¦\") sits right next to the tool_calls it produced. This is the record you\ndebug against.
Plant a realistic bug: the RAG API expects {\"question\": ...}, but it's natural to\nsend {\"query\": ...} because the tool's parameter is named query. One wrong key:
sed -i 's/{\"question\": query}/{\"query\": query}/' agent.py
docker compose run --rm --build agent
It crashes:
\n![\"The](\"https://wec.wiline.com/docs/assets/images/agent-422-crash-af97e2fe3ae5709ec3fe6d5f859095a0.png\")
\nFigure 6. The terminal gives you a stack trace ā it tells you where the code\nbroke. It does not tell you whether the agent thought correctly. For that, go to the\ntrace.
Open the failed run and click the red search_docs span:
![\"The](\"https://wec.wiline.com/docs/assets/images/trace-error-span-ddca621bf15a8cc64c96371cd6a42e1a.png\")
\nFigure 7. The whole diagnosis in one picture: call_model is green (the reasoning\nwas right ā it correctly chose to search), search_docs is red, died in 0.04s, and\nthe panel shows the 422 plus the exact payload it sent. An instant failure is a\nrejection, not a timeout ā pointing straight at a bad request.
Fix the one line and confirm green:
\nsed -i 's/{\"query\": query}/{\"question\": query}/' agent.py
docker compose run --rm --build agent
![\"The](\"https://wec.wiline.com/docs/assets/images/trace-fixed-green-c3b4677676dc40f861fca6dafe1594bb.png\")
\nFigure 8. Back to green ā search_docs succeeds and the final answer is produced. You\ndiagnosed and verified the fix from the trace.
That was the loud failure ā it crashed, so you'd have caught it eventually even without\na trace. The dangerous one is next.
\nReal agents have more than one tool, and the interesting decisions happen when the model\npicks between them. Add a get_pricing tool alongside search_docs, and route calls\nthrough a dispatch table keyed by tool name:
TOOLS = [
{\"type\": \"function\", \"function\": {
\"name\": \"search_docs\",
\"description\": \"Search the WEC documentation for how-to and setup questions.\",
\"parameters\": {\"type\": \"object\", \"properties\": {\"query\": {\"type\": \"string\"}}, \"required\": [\"query\"]}}},
{\"type\": \"function\", \"function\": {
\"name\": \"get_pricing\",
\"description\": \"Get the price of a WEC resource (e.g. 'compute instance', 'block storage', 'inference').\",
\"parameters\": {\"type\": \"object\", \"properties\": {\"resource\": {\"type\": \"string\"}}, \"required\": [\"resource\"]}}},
]
PRICES = {
\"compute instance\": \"$0.03 / vCPU-hour\",
\"block storage\": \"$0.10 / GB-month\",
\"inference\": \"$0.50 per 1M tokens\",
}
@observe()
def get_pricing(resource):
return PRICES.get(resource.lower().strip(), f\"No pricing found for '{resource}'.\")
TOOL_FUNCS = {\"search_docs\": search_docs, \"get_pricing\": get_pricing}
# in run(), dispatch by the tool the model actually chose:
for tc in msg[\"tool_calls\"]:
name = tc[\"function\"][\"name\"]
args = json.loads(tc[\"function\"][\"arguments\"])
print(f\"[action] {name}({args})\")
result = str(TOOL_FUNCS[name](**args))
messages.append({\"role\": \"tool\", \"tool_call_id\": tc[\"id\"], \"content\": result})
Ask a pricing question (run(\"How much does a compute instance cost on WEC?\")) and run\nit. The agent should pick get_pricing, not search_docs:
![\"Langfuse](\"https://wec.wiline.com/docs/assets/images/trace-two-tools-ba891f9c0ec4d99aea2e499dcbba3189.png\")
\nFigure 9. Genuine tool selection: given two tools, the agent chose get_pricing and\nanswered $0.03/vCPU-hour, Correctness 1.00. The choice itself is now a thing you can\nsee in the trace.
Now a bug that doesn't crash. Someone refactors the dispatch and hardcodes the old\nsingle tool, forgetting the new one ā so every call runs search_docs no matter what the\nmodel chose:
sed -i 's/result = str(TOOL_FUNCS\\[name\\](\\*\\*args))/result = str(search_docs(list(args.values())[0])) # BUG/' agent.py
docker compose run --rm --build agent
[action] get_pricing({'resource': 'compute instance'})
=== FINAL ANSWER ===
Based on the pricing information available, the highest cost compute instance category on
WEC is $1,586.77. ...
Read that carefully. The model chose get_pricing (correct). The answer is confident,\ndetailed ā and completely wrong (the real price is $0.03/vCPU-hour). There is no\nerror, no traceback. A log would look perfectly healthy. This is the failure mode that\nships to production and quietly lies to users.
The trace is the only thing that catches it. Open the run:
![\"The](\"https://wec.wiline.com/docs/assets/images/trace-silent-failure-162a3ce9394bad7caca0e908be683b67.png\")
\nFigure 10. Three tells a log can't show: (1) the executed span is search_docs ā\nthere's no get_pricing span, even though the model chose it: decision ā action;\n(2) the reasoning node scored Correctness 0.00 ā the LLM-judge auto-flagged the wrong\nanswer; (3) the output shows the model confused by a tool result that didn't match what\nit asked for. Fix the dispatch back to TOOL_FUNCS[name](**args) and the price is correct\nagain.
You instrumented a tool-calling agent so its reasoning and its actions are separate,\ninspectable trace nodes ā and debugged two real failures from the trace: the loud\ncrash and the silent, confident-but-wrong answer that no stack trace would ever reveal.
get_client().flush() before the process ends (already in v3).search_docs span missing. Only @observe-decorated functions become nodes ā\ndecorate the function, not the call site.usage_details=data.get(\"usage\") via\nupdate_current_generation; without it Langfuse can't compute cost.tool_calls (what it chose)\nagainst the executed spans (what actually ran). A mismatch is a dispatch/routing bug ā and\na correctness score on the trace will flag it automatically.You can now see an agent think, act, and fail quietly ā and catch it. Point the same\ninstrumentation at a real multi-step agent, add automated correctness scoring on every\ntrace, and you have production observability that surfaces silent failures before your users\ndo.
\ncd ~/agent-tracing && docker compose down --rmi local
![\"WhatsApp\"](\"https://wec.wiline.com/docs/assets/images/whatsapp_logo-cc34546fcff90d36128c7230c48c496f.png\"){kind=logo}
++![\"Evolution\"](\"https://wec.wiline.com/docs/assets/images/0_hT2ECga7Z67dEbYz-62f9ab60d339ae8868fee46475e505bc.png\")
Most \"self-host a WhatsApp AI\" guides stop at \"the container started.\" This one goes all\nthe way: you deploy a real, programmable WhatsApp gateway (Evolution API), then\nwrite the bridge yourself ā the ~50 lines that turn an incoming message into an LLM\nanswer and send it back. That bridge (webhook ā model ā reply) is the reusable pattern\nbehind every chat-AI integration: SMS, Slack, Telegram, voice ā swap the channel, the\nshape is identical.
\nAnd because this is a real build, we hit ā and fix ā every gotcha: an image that moved\npublishers, a Baileys version loop, an infinite reply loop, group-chat spam, WhatsApp's\nnew LID addressing, and a genuine delivery wall that most tutorials pretend\ndoesn't exist. Every command, error, and output below is from an actual run.
\n\nEvolution links to a WhatsApp number as a companion device (like WhatsApp Web) ā the\nbot acts as that account and can read every DM and group it receives. For a demo on a\nnumber you control it's fine; for anything real, use a dedicated number. And see the\ndelivery limitation at the end before you build on this in\nproduction.
Four containers: Evolution + its Postgres + Redis, and the bridge you'll write. The\nbridge is the whole point ā everything else is off-the-shelf.
\nPrerequisites: a WEC Instance with Docker + Compose, a WEC Inference API key\n(Inference ā API Keys), a WhatsApp number for\nthe demo, and ~2 GB free disk.
\nEvolution needs its own Postgres and Redis. We give it a dedicated set on an internal\nnetwork and expose only the API port (8080). docker-compose.yml:
services:
evolution-api:
image: evoapicloud/evolution-api:v2.2.3
restart: unless-stopped
ports:
- \"8080:8080\"
env_file: .env
volumes:
- evolution_instances:/evolution/instances
depends_on: [evolution-postgres, evolution-redis]
logging:
driver: json-file
options: { max-size: \"50m\", max-file: \"3\" }
evolution-postgres:
image: postgres:17
restart: unless-stopped
environment:
POSTGRES_USER: evolution
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: evolution
volumes:
- evolution_pgdata:/var/lib/postgresql/data
evolution-redis:
image: redis:7
restart: unless-stopped
volumes:
- evolution_redis:/data
volumes:
evolution_instances:
evolution_pgdata:
evolution_redis:
The logging: block caps container logs at 3Ć50 MB. Skip it and a chatty container can\nfill the disk and take the box down ā a lesson learned the hard way on this VM. Set it\nbefore you need it.
The .env (generated secrets, connection URIs, and ā importantly ā a pinned Baileys\nWhatsApp-Web version):
cd ~/evolution-api && cat > .env << EOF
AUTHENTICATION_API_KEY=$(openssl rand -hex 24)
SERVER_URL=http://<your-vm-ip>:8080
POSTGRES_PASSWORD=$(openssl rand -hex 16)
DATABASE_ENABLED=true
DATABASE_PROVIDER=postgresql
DATABASE_CONNECTION_URI=postgresql://evolution:\\${POSTGRES_PASSWORD}@evolution-postgres:5432/evolution?schema=public
CACHE_REDIS_ENABLED=true
CACHE_REDIS_URI=redis://evolution-redis:6379/0
CACHE_REDIS_PREFIX_KEY=evolution
CACHE_LOCAL_ENABLED=false
QRCODE_LIMIT=30
EOF
Bring it up:
\ndocker compose up -d
sleep 15
curl -s http://localhost:8080 | jq
{
\"status\": 200,
\"message\": \"Welcome to the Evolution API, it is working!\",
\"version\": \"2.2.3\",
...
}
![\"Evolution](\"https://wec.wiline.com/docs/assets/images/wa-evolution-up-680a7f5fe9e4a450cacf33f1df5ed6f7.png\")
\nFigure 1. The stack is up: Evolution answers on :8080 with its version and status.
(1) The publisher moved. The image most guides reference ā atendai/evolution-api ā\nnow returns \"pull access denied / repository does not exist.\" The current image is\nevoapicloud/evolution-api. The old repo's tag list still resolves, which sends you\ndown a rabbit hole; confirm with a fresh pull of hello-world that it's not rate-limiting.\n(2) Pin a real tag. v2.1.1 (from a popular blog) was never published ā check\nhttps://hub.docker.com/v2/repositories/evoapicloud/evolution-api/tags and pin one that\nexists (we use v2.2.3).
Set your API key as a shell var (every request needs it in an apikey: header):
APIKEY=$(grep '^AUTHENTICATION_API_KEY=' .env | cut -d= -f2)
Create the instance:
\ncurl -s -X POST http://localhost:8080/instance/create \\
-H \"apikey: $APIKEY\" -H \"Content-Type: application/json\" \\
-d '{\"instanceName\":\"wec-demo\",\"integration\":\"WHATSAPP-BAILEYS\",\"qrcode\":true}' | jq '.instance'
Then open the built-in Manager at http://<your-vm-ip>:8080/manager, enter your server\nURL + API key, click wec-demo, and scan the QR from WhatsApp ā Settings ā Linked\nDevices ā Link a Device. Confirm it's connected:
curl -s http://localhost:8080/instance/fetchInstances -H \"apikey: $APIKEY\" \\
| jq '.[] | {name, connectionStatus}'
{ \"name\": \"wec-demo\", \"connectionStatus\": \"open\" }
![\"Evolution](\"https://wec.wiline.com/docs/assets/images/wa-manager-connected-d91fda82b98c6238d0d747b47636645b.png\")
\nFigure 2. The Evolution Manager dashboard once the instance links: Connected, with live contact, chat, and message counts.
If the instance never leaves connecting and the logs show ChannelStartupService\nre-initializing every few seconds with a Baileys version env: 2,3000,... line, the\npinned WhatsApp-Web version is stale and WhatsApp rejects the handshake before a QR is\never generated. Fetch the current version and pin it, then recreate:
curl -s \"https://raw.githubusercontent.com/WhiskeySockets/Baileys/master/src/Defaults/baileys-version.json\"
# {\"version\":[2,3000,1035194821]}
echo \"CONFIG_SESSION_PHONE_VERSION=2.3000.1035194821\" >> .env
docker compose up -d --force-recreate evolution-api
Now the part you actually write. Rule: never wire logic before you've seen the real\ndata. So v1 does nothing but print whatever Evolution sends.
\nThe bridge runs as another container in the same Compose, so Evolution reaches it by\nname (http://bridge:8090) and the bridge reaches Evolution at\nhttp://evolution-api:8080. Create bridge/main.py:
from fastapi import FastAPI, Request
app = FastAPI()
@app.post(\"/webhook\")
async def webhook(request: Request):
data = await request.json()
print(\"=== WEBHOOK RECEIVED ===\", flush=True)
print(data, flush=True)
return {\"received\": True}
flush=True mattersWithout it, print output is buffered and never shows in docker compose logs ā you'll\nstare at an empty log convinced it's broken.
bridge/Dockerfile:
FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi uvicorn requests
COPY main.py .
CMD [\"uvicorn\", \"main:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8090\"]
Add the service to docker-compose.yml (inside services:), then build it:
bridge:
build: ./bridge
restart: unless-stopped
env_file: .env
depends_on: [evolution-api]
logging:
driver: json-file
options: { max-size: \"10m\", max-file: \"3\" }
docker compose up -d --build bridge
Now register the webhook so Evolution POSTs messages to the bridge:
\ncurl -s -X POST \"http://localhost:8080/webhook/set/wec-demo\" \\
-H \"apikey: $APIKEY\" -H \"Content-Type: application/json\" \\
-d '{\"webhook\":{\"enabled\":true,\"url\":\"http://bridge:8090/webhook\",\"webhookByEvents\":false,\"events\":[\"MESSAGES_UPSERT\"]}}' | jq '.enabled, .url'
Tail the logs (docker compose logs -f bridge) and send yourself a WhatsApp message. The\nreal payload appears:
{'event': 'messages.upsert', 'instance': 'wec-demo',
'data': {'key': {'remoteJid': '123456789012345@lid', 'fromMe': False, 'id': '...'},
'pushName': 'Test Contact',
'message': {'conversation': 'Oii'},
'messageType': 'conversation'},
'sender': '5511988887777@s.whatsapp.net', ...}
Three things we now know ā and every one drives the code next:
\ndata.message.conversation (plain) or data.message.extendedTextMessage.text (quoted)data.key.fromMe ā True for our own messages (the loop guard)remoteJid ends in @lid ā WhatsApp's privacy identifier, not a phone number (this one's a saga; Step 5)Give the bridge the WEC Inference key so it can call the model:
\necho \"WEC_API_KEY=<your-wec-inference-key>\" >> ~/evolution-api/.env
Now the logic ā main.py v2:
import os, requests
from fastapi import FastAPI, Request
app = FastAPI()
EVOLUTION_URL = \"http://evolution-api:8080\"
INSTANCE = \"wec-demo\"
EVOLUTION_APIKEY = os.environ[\"AUTHENTICATION_API_KEY\"]
WEC_API_KEY = os.environ[\"WEC_API_KEY\"]
WEC_URL = \"https://inference.wiline.com/v1/chat/completions\"
MODEL = \"Qwen2.5-3B-Instruct\"
def ask_llm(text: str) -> str:
r = requests.post(WEC_URL,
headers={\"Authorization\": f\"Bearer {WEC_API_KEY}\"},
json={\"model\": MODEL, \"messages\": [{\"role\": \"user\", \"content\": text}]},
timeout=60)
r.raise_for_status()
return r.json()[\"choices\"][0][\"message\"][\"content\"]
def send_whatsapp(number: str, text: str):
r = requests.post(f\"{EVOLUTION_URL}/message/sendText/{INSTANCE}\",
headers={\"apikey\": EVOLUTION_APIKEY, \"Content-Type\": \"application/json\"},
json={\"number\": number, \"text\": text}, timeout=60)
print(f\"SEND -> {r.status_code}\", flush=True)
@app.post(\"/webhook\")
async def webhook(request: Request):
data = (await request.json()).get(\"data\", {})
key = data.get(\"key\", {})
if key.get(\"fromMe\"): # (1) LOOP GUARD
return {\"skipped\": \"fromMe\"}
msg = data.get(\"message\", {}) # (2) extract text
text = msg.get(\"conversation\") or msg.get(\"extendedTextMessage\", {}).get(\"text\")
if not text:
return {\"skipped\": \"no text\"}
to = key.get(\"remoteJid\")
print(f\"IN {to}: {text}\", flush=True)
answer = ask_llm(text) # (3) think
send_whatsapp(to, answer) # (4) reply
print(f\"OUT {to}: {answer[:60]}...\", flush=True)
return {\"ok\": True}
The one line beginners always miss is (1) the loop guard. When the bridge sends a\nreply, that outgoing message fires another MESSAGES_UPSERT webhook with fromMe: true\nā without the guard, the bridge answers itself, forever. Rebuild:
docker compose up -d --build bridge
![\"The](\"https://wec.wiline.com/docs/assets/images/wa-bridge-logs-53cb3cf64a3d62d69794ee3a0bf3742b.png\")
\nFigure 3. v2 in action: a DM comes in, the model answers, the reply goes out.
It answers DMs now. But watch what happens next.
\nTail the logs with the bot live and you'll see it try to answer every group you're in,\nincluding promo/spam groups, and Evolution times out sending to them ā flooding errors:
\nIN 120363040812472138@g.us: *ALĆ CORREDORES š ... pechin.co/135449*
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='evolution-api', port=8080): Read timed out.
A companion account receives everything. Fix: answer DMs only ā groups end in\n@g.us:
to = key.get(\"remoteJid\", \"\")
if to.endswith(\"@g.us\") or \"broadcast\" in to: # DMs only
return {\"skipped\": \"not a DM\"}
Now the subtler wall. With groups filtered, a real DM comes in ā but the reply gets\nrejected:
\n{\"status\":400,\"error\":\"Bad Request\",\"response\":{\"message\":[
{\"exists\":false,\"jid\":\"123456789012345@lid\",\"name\":\"Test Contact\",\"number\":\"123456789012345@lid\"}]}}
![\"Evolution](\"https://wec.wiline.com/docs/assets/images/wa-lid-400-84cc0430abac7bdd5aa6d29cf1c100b5.png\")
\nFigure 4. The LID wall in the logs: the incoming DM arrives as @lid, and the reply is rejected with exists:false.
exists: false. This is WhatsApp's LID addressing (a 2025 privacy change): incoming\nDMs arrive with an @lid identifier, and you cannot send back to an @lid ā Evolution\nneeds the real @s.whatsapp.net phone JID. Evolution's contact store has both, though:
curl -s -X POST \"http://localhost:8080/chat/findContacts/wec-demo\" \\
-H \"apikey: $APIKEY\" -H \"Content-Type: application/json\" -d '{}' \\
| jq '[.[] | select(.pushName==\"Test Contact\")]'
[ { \"remoteJid\": \"123456789012345@lid\", \"pushName\": \"Test Contact\" },
{ \"remoteJid\": \"5511999999999@s.whatsapp.net\", \"pushName\": \"Test Contact\" } ]
![\"findContacts](\"https://wec.wiline.com/docs/assets/images/wa-findcontacts-275c01bee94ec0da990f2cdd0ec6acf2.png\")
\nFigure 5. Evolution stores both identities for the same person ā the @lid and the real phone JID. That is the key to resolving the reply address.
So we resolve the @lid to the phone number before replying. Add a resolver and use it:
def resolve_number(remote_jid: str, push_name: str):
if remote_jid.endswith(\"@s.whatsapp.net\"):
return remote_jid.split(\"@\")[0]
# LID: find the contact's real number by matching the name
contacts = requests.post(f\"{EVOLUTION_URL}/chat/findContacts/{INSTANCE}\",
headers={\"apikey\": EVOLUTION_APIKEY, \"Content-Type\": \"application/json\"},
json={}, timeout=30).json()
for c in contacts:
if c.get(\"pushName\") == push_name and c.get(\"remoteJid\", \"\").endswith(\"@s.whatsapp.net\"):
return c[\"remoteJid\"].split(\"@\")[0]
return None
In the handler, resolve before sending:
\n number = resolve_number(to, data.get(\"pushName\", \"\"))
if not number:
return {\"skipped\": \"unresolved lid\"}
...
send_whatsapp(number, answer)
Now SEND -> 201. (Matching by name is a heuristic ā two contacts sharing a name would\ncollide; production keeps a proper LIDāphone map. But it works, and it's the honest state\nof Evolution's LID support in v2.2.3.)
LLMs emit Markdown (**bold**, [text](url)), but WhatsApp has its own formatting:\nbold is *single asterisk*, and Markdown links don't render. So **Compute** shows up\nwith literal asterisks. Translate the model's output before sending, and append the RAG's\nsources as plain URLs:
import re
def to_whatsapp(md: str) -> str:
md = re.sub(r\"\\*\\*(.+?)\\*\\*\", r\"*\\1*\", md) # **bold** -> *bold*
md = re.sub(r\"\\[([^\\]]+)\\]\\((https?://[^)]+)\\)\", r\"\\1 (\\2)\", md) # [t](url) -> t (url)
md = re.sub(r\"(?m)^#{1,6}\\s*\", \"\", md) # drop # headings
return md
Verified against a real answer ā exactly what WhatsApp will render:
\nTo create a compute instance on WiLine Edge Cloud (WEC), follow these steps:
1. Log in to the WiLine Edge Cloud.
2. In the sidebar, click *Compute*.
3. Select *Instances*.
4. Click the \"Launch Virtual Machine\" wizard to begin the deployment process.
_Sources:_
https://wec.wiline.com/docs/cloud_portal/platform/compute/instances/compute_instance/
![\"The](\"https://wec.wiline.com/docs/assets/images/wa-formatted-output-766cfd620ca72685d8f37110da283d56.png\")
\nFigure 6. The model output after to_whatsapp(): single-asterisk bold, plain links, sources appended.
Everything so far uses the raw model, so a WEC question gets a general answer. To make it\nanswer from your documentation, swap ask_llm to call the RAG service from the\nevals series ā one function, nothing\nelse changes:
RAG_URL = \"http://<your-vm-ip>:8000/ask\"
def ask_llm(text: str) -> str:
r = requests.post(RAG_URL, json={\"question\": text}, timeout=120)
r.raise_for_status()
d = r.json()
answer = to_whatsapp(d[\"answer\"])
sources = d.get(\"sources\", [])[:2]
if sources:
answer += \"\\n\\n_Sources:_\\n\" + \"\\n\".join(sources)
return answer
That's the payoff of the bridge pattern: swap the brain, keep the plumbing. Now WhatsApp\nanswers WEC questions from the real docs, with sources:
\n![\"WhatsApp](\"https://wec.wiline.com/docs/assets/images/wa-bridge-reply-bf0eaa9a06721ee1396cbf44fd96ae3a.png\")
\nFigure 7. A real message in ā the bridge ā RAG over the WEC docs ā a grounded answer\ndelivered on WhatsApp, sources included.
You self-hosted a programmable WhatsApp gateway and wrote the bridge that turns it into\nan AI assistant ā webhook in, model out, reply back ā grounded in your own docs.
Here's the part other tutorials won't tell you. Even with SEND -> 201, replies to a\nLID-migrated account frequently arrive as \"Waiting for this message\" on the\nrecipient's phone ā WhatsApp accepted the ciphertext but no device can decrypt it. It's a\nknown Baileys companion-device limitation, and WhatsApp's LID rollout makes it worse.
What we observed, honestly:
\n\"Waiting for this message\" returns ā even though\nevery send still reports 201.So the bridge is correct end-to-end; the unofficial Baileys transport is the weak link.\nIf you're building this for real:
\nEvolution shines for outbound automation to numbers you control (notifications, alerts,\nflows). As a two-way AI assistant on a personal LID account, treat delivery as best-effort.
\npull access denied on atendai/evolution-api ā image moved to evoapicloud/evolution-api.connecting, re-init loop ā stale Baileys version; pin CONFIG_SESSION_PHONE_VERSION.fromMe loop guard.@g.us; answer DMs only.SEND -> 400 exists:false ...@lid ā resolve the LID to the phone JID via findContacts.**asterisks** in replies ā translate Markdown to WhatsApp formatting.cd ~/evolution-api && docker compose down # add -v to also wipe the DB/session
Unlink the device in WhatsApp ā Linked Devices if you're done.
\nYou built the reusable bridge pattern ā webhook ā resolve ā model ā reply. Point it at a\ndifferent brain, a different channel, or add tools. If you want an AI agent on WhatsApp\nwithout the Baileys caveats, the OpenClaw native WhatsApp\nchannel is the managed path; for docs-grounded\nanswers, the evals & observability series\nbuilds the RAG service this tutorial plugs into.
", "url": "https://wec.wiline.com/docs/tutorials/whatsapp-ai-assistant-evolution-api/", "title": "Build a WhatsApp AI assistant from scratch with Evolution and the WEC API", "summary": "Self-host a programmable WhatsApp gateway (Evolution API) and write the ~50-line bridge that turns it into an AI assistant ā webhook in, LLM out, reply back. Every gotcha is real: the image that moved publishers, the Baileys version loop, the loop guard, group spam, WhatsApp's LID addressing, and the delivery wall nobody warns you about.", "date_modified": "2026-07-21T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "self-hosting", "whatsapp", "evolution-api", "docker", "inference" ] }, { "id": "https://wec.wiline.com/docs/tutorials/add-whatsapp-channel-openclaw/", "content_html": "![\"OpenClaw\"](\"https://wec.wiline.com/docs/assets/images/openclaw-wordmark-003352a1a7f02afc3bd877ae6f2dc175.png\")
+Telegram (Part 3) gave your agent a bot.\nWhatsApp gives it a phone line ā the app ~3 billion people already use, reachable with\nzero friction. One catch worth understanding up front: WhatsApp has no bot account, so\nOpenClaw links to a real number as a companion device (like WhatsApp Web) and the\nagent acts as that account. Every command and error below is from a real run.
\n\nAnyone messages the number; the gateway ā linked as a companion device ā runs it through\nyour model and replies from the number itself. No inbound ports, no bot account.
\n\nPrerequisites: Parts 1ā4 done (OpenClaw in ~/openclaw), a WhatsApp account on your\nphone, ~1 GB free disk. All commands run in ~/openclaw.
OpenClaw treats messaging apps as channels; WhatsApp is a first-class one:
\ndocker compose run --rm openclaw-cli channels add --channel whatsapp
Installed WhatsApp plugin
Added WhatsApp account \"default\".
requires plugin API >=2026.7.1? You're on an older OpenClaw ā upgrade first.The WhatsApp channel is a plugin gated to newer runtimes:
![\"The](\"https://wec.wiline.com/docs/assets/images/wa-version-wall-db236509ca6e250b19147327473e0932.png\")
\nFigure 1. The version wall ā the WhatsApp plugin needs OpenClaw ā„2026.7.1.
Back up, pull, recreate, and confirm your existing channels survived (:latest does\nnot auto-update ā you must pull deliberately):
cp ~/.openclaw/openclaw.json ~/.openclaw/openclaw.json.bak-$(date +%Y%m%d)
docker compose pull openclaw-gateway
docker compose run --rm openclaw-cli --version # expect 2026.7.1+
docker compose up -d --force-recreate openclaw-gateway
docker compose run --rm openclaw-cli channels status # Telegram should still be connected
![\"The](\"https://wec.wiline.com/docs/assets/images/wa-version-check-b1b98be0f2990d46e5686130b2722d3b.png\")
\nFigure 2. After pulling, the image reports 2026.7.1 ā the requirement is met.
Then re-run the channels add above.
Check status and the channel is stopped with a cryptic error:
\n![\"WhatsApp](\"https://wec.wiline.com/docs/assets/images/wa-trust-error-7a8033b008450d82190f783b79998f71.png\")
\nFigure 3. stopped ā an untrusted plugin canāt use openKeyedStore, so the channel wonāt start.
That's a security feature: OpenClaw sandboxes third-party plugins, and an untrusted\none can't use openKeyedStore (the storage WhatsApp needs for its session). Trust it ā\ninclude any other non-bundled plugin already loading (here codex) so you don't lock it\nout ā and recreate:
docker compose run --rm openclaw-cli config set \\
--batch-json '[{\"path\":\"plugins.allow\",\"value\":[\"codex\",\"whatsapp\"]}]'
docker compose up -d --force-recreate openclaw-gateway
docker compose run --rm openclaw-cli channels status
![\"plugins.allow](\"https://wec.wiline.com/docs/assets/images/wa-trust-fix-503842c5815cf95dc4bde07dc3b4a6ee.png\")
\nFigure 4. Trusted ā plugins.allow now lists whatsapp (and codex).
The restart matters ā the gateway reads plugins.allow before it initializes the\nplugin. Now it's up:
![\"WhatsApp](\"https://wec.wiline.com/docs/assets/images/wa-connected-d4ec6a75486c355d24570487f62305c8.png\")
\nFigure 5. Live ā WhatsApp running, connected, healthy, and Telegram still up.
docker compose run --rm openclaw-cli channels login
Scan the terminal QR from WhatsApp ā Settings ā Linked Devices ā Link a Device.
\n![\"The](\"https://wec.wiline.com/docs/assets/images/wa-qr-544d7332e052ce0469c77d7e60fec42a.png\")
\nFigure 6. Scan to link; ā
Linked after restart handles the code 515 reconnect.
code 515 restart is normalWhatsApp forces a reconnect right after pairing; OpenClaw handles it automatically. This\nis exactly where hand-rolled setups get stuck in a loop ā here it just works.
From another phone, message the number:
\n![\"The](\"https://wec.wiline.com/docs/assets/images/wa-reply-08f4125cb353af3cff9a2418928be7ec.png\")
\nFigure 7. A question in, the agentās answer back ā sent from the number, as the account.
It replies ā but notice it answers from your number, as you. That's companion mode:\nthe agent is the account. Great for a personal assistant, and the reason for the warning\nbelow.
\nYour OpenClaw agent answers on WhatsApp ā natively, no bridge, no bot account.
It processes every DM and group the account receives (Listening for ... DM + all groups). For anything real: link a dedicated number (a cheap prepaid SIM ā virtual\nnumbers are often rejected), and set an allowlist before exposing it.
Park the channel when you're done testing (Telegram keeps running; your link stays saved):
\ndocker compose run --rm openclaw-cli config set \\
--batch-json '[{\"path\":\"channels.whatsapp.accounts.default.enabled\",\"value\":false}]'
docker compose up -d --force-recreate openclaw-gateway
openKeyedStore is only available for trusted plugins ā channel installs but stays\nstopped. Add the id to plugins.allow and recreate the gateway (Step 2). A live\nconfig change isn't enough once it has already failed to start.code 515 ā normal post-pairing restart. If it never settles, remove\nstale entries in WhatsApp ā Linked Devices and scan a fresh QR.Part 5 done ā your agent reaches Telegram and WhatsApp from a private, hardened box.
\ndocker compose run --rm openclaw-cli channels remove --channel whatsapp
docker compose restart openclaw-gateway
And unlink the device in WhatsApp ā Linked Devices if you're done.
", "url": "https://wec.wiline.com/docs/tutorials/add-whatsapp-channel-openclaw/", "title": "Add a WhatsApp channel to OpenClaw", "summary": "Put your self-hosted agent on WhatsApp. Add the channel, trust the plugin, scan a QR ā and understand why a companion link makes the agent act as your account. Every command and error from a real run.", "date_modified": "2026-07-20T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "self-hosting", "openclaw", "whatsapp", "chatbot" ] }, { "id": "https://wec.wiline.com/docs/tutorials/deepeval-regression-testing-wiline-inference/", "content_html": "Our GPT-5.6 token-economics analysis ended with a challenge:\nbenchmark promises are a hypothesis, not a reason ā run the eval on your workload before you\nbelieve a \"fewer tokens per task\" claim. This tutorial is us taking our own advice.
\nWe take the RAG docs assistant from part 5,\nwrap it in a DeepEval regression suite ā containerized, judged by gemma4 on the WEC\nInference API, zero OpenAI dependency ā and then use that suite to answer a genuinely open\nquestion: should we swap the service's generation model? Qwen2.5-3B-Instruct (current) vs\ngemma4 (candidate), pass rate and tokens per task, measured head to head.
Spoiler: the eval saved us from a pointless migration. And along the way we hit four real\nproduction failures ā a disk-full death spiral, a startup race condition, a Docker Compose\nvariable trap that silently evaluated the wrong model, and the fix that makes that class of\nbug impossible. Every command, number, and error below is from a real run.
\n\nThree design decisions up front:
\ndocker compose run eval in both places. (Our first attempt was a\nvenv. It died to a full disk before installing anything; see Troubleshooting.)tests.csv embeds each\nreference fact inside an llm-rubric string ā one regex pulls it back out. Your eval data\noutlives your eval framework.rag-api answering on :8000, eval/tests.csv present).df -h / now. Ours read 100% and the story of why\nis in Troubleshooting.Never eval a service you haven't poked by hand first:
\ncurl -s -X POST localhost:8000/ask \\
-H \"Content-Type: application/json\" \\
-d '{\"question\": \"How do I create a compute instance on WEC?\"}' | python3 -m json.tool
![\"The](\"https://wec.wiline.com/docs/assets/images/deepeval-smoke-test-98290b8dda1f2a3f5e57a82fa43c2dfa.png\")
\nFigure 1. The system under test, alive: /ask answers from the WEC docs corpus and cites\nits sources. Never eval a service you haven't poked by hand first.
eval/Dockerfile:
FROM python:3.12-slim
WORKDIR /eval
RUN pip install --no-cache-dir deepeval==3.*
# Suite code is mounted at runtime so edits don't need a rebuild
CMD [\"deepeval\", \"test\", \"run\", \"test_rag_regression.py\"]
Add the service to docker-compose.yml inside the services: block (part 5's file ends\nwith a top-level volumes: section ā appending blindly puts your service in the wrong block;\ndocker compose config --quiet catches it):
eval:
build: ./eval
profiles: [\"eval\"]
env_file: .env
environment:
RAG_API_URL: http://rag-api:8000
volumes:
- ./eval:/eval
depends_on:
- rag-api
profiles: [\"eval\"] keeps it out of a normal docker compose up ā it only runs when you ask.\nBuild it:
docker compose --profile eval build eval
![\"docker](\"https://wec.wiline.com/docs/assets/images/deepeval-eval-build-58810296ef048e28edf1a9f90735d2eb.png\")
\nFigure 2. The eval runner builds in ~42s: python:3.12-slim plus DeepEval 3.x. This exact\nimage is what CI runs later ā no venv, no drift.
DeepEval's LLM-as-judge metrics want an OpenAI key. We don't have one and don't want one.\neval/wec_judge.py:
\"\"\"Custom DeepEval judge backed by WiLine Inference (OpenAI-compatible).\"\"\"
import os
from openai import OpenAI
from deepeval.models.base_model import DeepEvalBaseLLM
WEC_BASE_URL = os.getenv(\"WEC_BASE_URL\", \"https://inference.wiline.com/v1\")
class WECJudge(DeepEvalBaseLLM):
def __init__(self, model: str | None = None):
self.model_name = model or os.getenv(\"JUDGE_MODEL\", \"gemma4\")
self.client = OpenAI(
base_url=WEC_BASE_URL,
api_key=os.environ[\"WEC_API_KEY\"],
)
def load_model(self):
return self.client
def generate(self, prompt: str) -> str:
response = self.client.chat.completions.create(
model=self.model_name,
messages=[{\"role\": \"user\", \"content\": prompt}],
temperature=0,
)
return response.choices[0].message.content
async def a_generate(self, prompt: str) -> str:
return self.generate(prompt)
def get_model_name(self):
return f\"WEC/{self.model_name}\"
Prove it answers from inside the compose network before building anything on it:
\ndocker compose --profile eval run --rm eval \\
python -c \"from wec_judge import WECJudge; j = WECJudge(); print('judge says:', j.generate('Reply with exactly: WEC judge online'))\"
judge says: WEC judge online
![\"Judge](\"https://wec.wiline.com/docs/assets/images/deepeval-judge-online-c5844beb37c759e19f698f4ea927ec8c.png\")
\nFigure 3. \"WEC judge online\" ā gemma4 answering through the container, on the same compose\nnetwork as the service it will judge. No OpenAI key anywhere in the stack.
An honest caveat we carry through the rest of this tutorial: in the model comparison later,\ngemma4 is both judge and contestant. The mitigation is structural ā the judge only compares\nanswer-vs-reference-fact, never \"which model wrote this\" ā and we watch its verdicts on the\nknown-hard cases for favoritism. (It showed none: every case it failed gemma4 on is a case it also fails Qwen on.)
\nYou can point DeepEval's LLM-judged metrics at any OpenAI-compatible endpoint. Your eval\nverdicts ā questions, answers, reasoning ā never leave your infrastructure.
Part 5's /ask returned answer and sources ā and threw away the token usage that came back\nwith every completion. For token economics we need it, and for the A/B we need the model to be\nswitchable. Three env vars and two response fields in app/main.py (volume-mounted; uvicorn\nhot-reloads, no rebuild):
# Generation model is configurable so the eval suite can A/B models.
# Defaults preserve the tutorial-5 setup.
GEN_MODEL = os.getenv(\"GEN_MODEL\", \"Qwen2.5-3B-Instruct\")
GEN_BASE_URL = os.getenv(\"GEN_BASE_URL\", \"https://inference.wiline.com/v1\")
GEN_API_KEY = os.getenv(\"GEN_API_KEY\", os.environ[\"WEC_API_KEY\"])
generate() now also returns resp.usage, and /ask reports both:
{
\"answer\": \"To view your billing statements ...\",
\"sources\": [\"...\"],
\"model\": \"Qwen2.5-3B-Instruct\",
\"usage\": {
\"prompt_tokens\": 1238,
\"completion_tokens\": 122,
\"total_tokens\": 1360
}
}
![\"/ask](\"https://wec.wiline.com/docs/assets/images/deepeval-ask-usage-b1a5b036ae1af43fa37d1c91c9d2f092.png\")
\nFigure 4. The new model and usage fields: 1,238 prompt + 122 completion tokens for one\nbilling question. This is the number the token-economics debate is actually about.
That usage block is the number the GPT-5.6 marketing is about ā measured on your workload,\nper request. Pass the variable through compose (GEN_MODEL: ${GEN_MODEL:-Qwen2.5-3B-Instruct}\nunder rag-api's environment:) and remember this line ā it comes back to bite us in\nTroubleshooting.
eval/test_rag_regression.py. It reads part 5's tests.csv, extracts the reference fact from\neach promptfoo rubric, asks the live service, logs token usage, and asserts a GEval correctness\nmetric judged by WECJudge:
\"\"\"DeepEval regression suite for the WEC docs RAG service.
Reuses the promptfoo golden set (tests.csv) from the previous tutorial:
the reference fact is parsed out of each llm-rubric string.
\"\"\"
import csv, json, os, re
import pytest
import requests
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from wec_judge import WECJudge
RAG_API_URL = os.getenv(\"RAG_API_URL\", \"http://rag-api:8000\")
USAGE_LOG = os.getenv(\"USAGE_LOG\", \"usage_log.jsonl\")
MAX_CASES = int(os.getenv(\"MAX_CASES\", \"0\")) # 0 = all
FACT_RE = re.compile(r'Reference fact from the official docs: \"(.*?)\"')
def load_cases():
with open(\"tests.csv\", newline=\"\") as f:
for row in csv.DictReader(f):
m = FACT_RE.search(row[\"__expected\"])
yield row[\"question\"], (m.group(1) if m else \"\")
CASES = list(load_cases())
if MAX_CASES:
CASES = CASES[:MAX_CASES]
judge = WECJudge()
correctness = GEval(
name=\"Correctness\",
criteria=(
\"The actual output must correctly address the input question and must not \"
\"contradict the reference fact given as expected output. Different wording \"
\"is fine. Fail only if the answer is wrong, contradicts the reference fact, \"
\"or fails to answer the question.\"
),
evaluation_params=[
LLMTestCaseParams.INPUT,
LLMTestCaseParams.ACTUAL_OUTPUT,
LLMTestCaseParams.EXPECTED_OUTPUT,
],
model=judge,
threshold=0.5,
)
@pytest.mark.parametrize(
\"question,reference\", CASES, ids=[q[:50] for q, _ in CASES]
)
def test_rag_regression(question, reference):
r = requests.post(f\"{RAG_API_URL}/ask\", json={\"question\": question}, timeout=120)
r.raise_for_status()
data = r.json()
expect = os.getenv(\"EXPECT_MODEL\")
assert not expect or data[\"model\"] == expect, (
f\"suite expected {expect} but service is running {data['model']}\"
)
with open(USAGE_LOG, \"a\") as f:
f.write(json.dumps({
\"model\": data.get(\"model\"),
\"question\": question,
**data.get(\"usage\", {}),
}) + \"\\n\")
assert_test(
LLMTestCase(
input=question,
actual_output=data[\"answer\"],
expected_output=reference,
),
[correctness],
)
(That EXPECT_MODEL assertion wasn't in the first version. It exists because of a bug you'll\nmeet in a moment.)
First run, capped at 5 cases to fail fast:
\ndocker compose --profile eval run --rm \\
-e MAX_CASES=5 -e DEEPEVAL_TELEMETRY_OPT_OUT=YES \\
eval
4 of 5 passed in 2:37 ā and both the passes and the failure are worth reading. The\nfailure (\"What is the starting point for billing?\", score 0.3) isn't a service bug: the\nanswer was factually fine, but the golden label is a section title (\"How Billing Works\")\nand the judge wanted it named. Remember this case ā it flickers between pass and fail all\ntutorial long, and that flicker becomes a finding. Meanwhile on \"total number of events\"\nthe judge scored a 0.6 borderline pass with exactly the right reasoning (the service\ndescribed the feature instead of giving steps). Nuanced verdicts from a self-hosted judge ā\nthe thing people assume you need GPT-4-class models for.
\n![\"First](\"https://wec.wiline.com/docs/assets/images/deepeval-first-run-cabfff276f20b525d0c7c20bb52c1a6a.png\")
\nFigure 5. First run: 4 of 5 passed in 2:37, every verdict with written reasoning from\nthe gemma4 judge ā including the one failure, a strict verdict on a section-title label\nrather than a service bug.
Note DeepEval's summary line: token cost: None. It can't price a self-hosted judge ā which is\nexactly why the suite writes its own usage_log.jsonl.
![\"Per-question](\"https://wec.wiline.com/docs/assets/images/deepeval-usage-log-65042f9bd05652b9ff3814cd52b37270.png\")
\nFigure 6. The suiteās own accounting: one JSON line per question ā model, prompt, completion,\ntotal ā the raw data every comparison in this tutorial is built from.
You have a regression suite grading a live RAG service ā reusing last tutorial's golden set\nunchanged, judged by a self-hosted model, with per-question token accounting the framework\nitself doesn't provide.
The question the suite exists to answer: our service generates with Qwen2.5-3B-Instruct ā\nshould it? gemma4 is newer, well-regarded, already in the WEC catalog. Feels like an upgrade.\nFeels. Let's measure.
\nBaseline, all 16 cases:
\ndocker compose --profile eval run --rm \\
-e DEEPEVAL_TELEMETRY_OPT_OUT=YES \\
-e USAGE_LOG=usage_qwen.jsonl \\
eval
14/16 (87.5%) in 8:50. The two failures are label artifacts, not RAG bugs: the golden facts\nfrom part 5 are section titles (\"How Billing Works\"), and the judge dinged answers that were\nfactually fine but didn't echo the title; and mp3 vs the label audio/mpeg ā the same fact at\ntwo abstraction levels, ruled a contradiction. Read your failures before trusting your pass\nrate. We leave both in: they're honest label debt, and they become our judge-favoritism canary\nin the A/B.
![\"Qwen](\"https://wec.wiline.com/docs/assets/images/deepeval-qwen-baseline-5b61265199e0d615241bbbfdb30d7a81.png\")
\nFigure 7. The baseline: 87.5% in 8:50. Both failures are label artifacts, not RAG bugs ā\nread your failures before trusting your pass rate.
Challenger ā same suite, one env var (prefix it on every compose command; Troubleshooting\nexplains why in blood):
\nGEN_MODEL=gemma4 docker compose up -d rag-api
until curl -sf localhost:8000/health > /dev/null; do echo \"waiting...\"; sleep 3; done
curl -s -X POST localhost:8000/ask -H \"Content-Type: application/json\" -d '{\"question\": \"ping\"}' \\
| python3 -c \"import json,sys; print('>>> model is:', json.load(sys.stdin)['model'])\"
GEN_MODEL=gemma4 docker compose --profile eval run --rm \\
-e DEEPEVAL_TELEMETRY_OPT_OUT=YES \\
-e USAGE_LOG=usage_gemma4.jsonl \\
-e EXPECT_MODEL=gemma4 \\
eval
And verify the log, because run labels lie:
\nsort <(cat eval/usage_gemma4.jsonl | python3 -c \"import json,sys; [print(json.loads(l)['model']) for l in sys.stdin]\") | uniq -c
# 16 gemma4
![\"Switching](\"https://wec.wiline.com/docs/assets/images/deepeval-gemma4-run-6a7c64183ec8b68e6d5e5ff76dd0924a.png\")
\nFigure 8. Switching the challenger in: GEN_MODEL=gemma4, wait for /health, then confirm\n>>> model is: gemma4 from the response itself ā before spending a single eval minute.
![\"gemma4](\"https://wec.wiline.com/docs/assets/images/deepeval-gemma4-verified-8d7291e9dfcda2f337f83ca7e4988071.png\")
\nFigure 9. The gemma4 run: 81.25% in ~13 minutes ā and its three failures are the same\nlabel-artifact cases Qwen flickers on. No favoritism from the gemma4 judge toward the gemma4\ncontestant.
One script over the usage logs:
\n| run | pass rate | avg prompt tokens | avg completion tokens | total tokens |
|---|---|---|---|---|
| Qwen2.5-3B (run 1) | 87.5% | 1,126 | 75 | 19,225 |
| Qwen2.5-3B (run 2) | 81.25% | 1,126 | 75 | 19,225 |
| gemma4 | 81.25% | 1,199 | 415 | 25,834 |
![\"Token](\"https://wec.wiline.com/docs/assets/images/deepeval-verdict-table-fc9c10d332e08f09d63c768e48c1b23c.png\")
\nFigure 10. The verdict in five lines: same pass rate, 5.5Ć the completion tokens. The eval\njust prevented a pointless migration.
Two findings, one expected and one not:
\nThe verdict: don't switch. Same effective quality, same failure cases ā but gemma4 uses\n5.5Ć the completion tokens (415 vs 75 per answer) and runs ~45% slower per question. On a\nRAG workload where the ~1,100-token retrieved context dominates the prompt, the completion\ncolumn is the whole cost difference. \"Newer model\" lost to \"measure it\" in one afternoon. This\nis precisely the eval the GPT-5.6 news post said to run ā and it's the same eval you'd run\nagainst GPT-5.6 itself: change GEN_MODEL, GEN_BASE_URL, and GEN_API_KEY, nothing else.
The judge has a noise floor. Look at the two Qwen rows: identical token counts ā\n19,225 in both runs, because temperature-0 generation is deterministic ā yet 87.5% vs 81.25%.\nThe answers didn't change; only the judge's verdicts did (one borderline 0.6 became a 0.3).\nThat cleanly isolates judge nondeterminism from model behavior, and it sets your CI threshold:\na gate at \"ā„ 87.5%\" would flake. Ours tolerates one flipped verdict and fails on two or more\nreal regressions.
\n![\"Qwen](\"https://wec.wiline.com/docs/assets/images/deepeval-qwen-run2-48156c484cd782d1e346e0938b1080b9.png\")
\nFigure 11. Qwen run 2: the same 16 deterministic answers, 81.25% this time ā the judge, not\nthe model, is where the variance lives.
You can settle a model-swap debate with measured data ā pass rate and tokens per task ā and\nyou know your judge's noise floor, so you can set a CI threshold that won't flake.
The payoff of containerizing everything: the CI job is the command you've been running all\nalong. .github/workflows/eval.yml:
name: rag-regression-eval
on:
pull_request:
paths: [\"app/**\", \"eval/**\"]
workflow_dispatch:
jobs:
eval:
runs-on: [self-hosted, wec] # runner with access to inference.wiline.com
steps:
- uses: actions/checkout@v4
- name: Run regression suite
env:
WEC_API_KEY: ${{ secrets.WEC_API_KEY }}
run: |
docker compose up -d rag-api
docker compose --profile eval run --rm \\
-e DEEPEVAL_TELEMETRY_OPT_OUT=YES \\
-e EXPECT_MODEL=Qwen2.5-3B-Instruct \\
eval
- name: Teardown
if: always()
run: docker compose down
Any PR that touches the app or the eval must survive 16 judged questions against the live\nservice ā with the model identity asserted ā before it merges.
\nYour eval is a CI gate. The container you ran locally is byte-for-byte what CI runs ā no\n\"works on my machine\" gap between development and the pipeline.
[Errno 28] No space left on device ā and it came backāOur first pip install deepeval died: disk 100% full, 58G of 58G. We freed 2 GB of journal\nlogs and build cacheā¦ and it was full again minutes later. The culprit:
7.9G /var/lib/docker/containers/10281d.../10281d...-json.log
langfuse-clickhouse-1 was in an error loop, spraying stack traces into a Docker json-log with\nno rotation configured ā a death spiral, because a full disk causes more errors causes more\nlog. It had been \"unhealthy\" for 8 days. The fix, in order (order matters ā truncating while\nthe container runs just refills):
docker stop langfuse-clickhouse-1
sudo sh -c \"truncate -s 0 /var/lib/docker/containers/10281d*/*-json.log\"
Then make it permanent ā log rotation in the compose override, applied to every service:
\nx-logging: &default-logging
logging:
driver: json-file
options:
max-size: \"50m\"
max-file: \"3\"
services:
clickhouse:
<<: *default-logging
# ... every other service
docker compose up -d to recreate. ClickHouse went healthy for the first time in 8 days.\nDocker's json-file driver does not rotate by default; on a box running LLM infrastructure\nthat logs enthusiastically, unbounded container logs are a time bomb.
Connection refused, then 3 passāChanging GEN_MODEL recreates rag-api, and the app takes ~3 minutes to boot (Chroma +\nembedding model load). Our suite started hammering /ask immediately: 13 straight\nconnection-refused failures, then the API came alive mid-run and the last 3 passed. Flaky in\nthe worst way ā the failure depends on wall-clock timing.
Fix it in the suite, not in human discipline ā eval/conftest.py:
\"\"\"Refuse to start the suite until the RAG API is healthy.\"\"\"
import os, time
import requests
RAG_API_URL = os.getenv(\"RAG_API_URL\", \"http://rag-api:8000\")
READY_TIMEOUT = int(os.getenv(\"READY_TIMEOUT\", \"300\"))
def pytest_configure(config):
deadline = time.time() + READY_TIMEOUT
while time.time() < deadline:
try:
requests.get(f\"{RAG_API_URL}/health\", timeout=3).raise_for_status()
return
except requests.RequestException:
time.sleep(3)
raise RuntimeError(f\"rag-api not healthy after {READY_TIMEOUT}s ā aborting eval\")
The nastiest one, because nothing errored. ${GEN_MODEL:-Qwenā¦} in compose is interpolated\nfrom your shell at every invocation. We started the service with\nGEN_MODEL=gemma4 docker compose up -d ā correct. Then ran the eval without the prefix.\nCompose resolved the variable back to the Qwen default, saw the running container as config\ndrift, and ā via depends_on ā silently recreated rag-api with Qwen. The run completed,\nthe summary said \"gemma4\" in every verdict (the judge was gemma4), and every answer had been\nwritten by Qwen.
We only caught it because the suite logs the model from the response:
\nsort <(... usage_gemma4.jsonl ...) | uniq -c
# 16 Qwen2.5-3B-Instruct ā the \"gemma4\" run
Never trust the label on a run; verify from data the system under test reports about itself.\nThe permanent fix is the EXPECT_MODEL assertion in the suite (Step 4). Proof it works ā the\ndeliberate-failure test:
FAILED ... AssertionError: suite expected gemma4 but service is running Qwen2.5-3B-Instruct
1 failed in 2.44s
Under three seconds, before spending a single judge call ā versus the 10+ minutes each mislabeled run\ncost us.
\n![\"EXPECT_MODEL](\"https://wec.wiline.com/docs/assets/images/deepeval-expect-model-ccc34be3d936ecfd93823c565073b405.png\")
\nFigure 12. The guard in action: wrong model ā loud failure in under three seconds, before a\nsingle judge call is spent. Compare with the 10+ minutes each silently mislabeled run cost us.
sleep 5 is not a readiness checkāBonus small one: after recreating the container we piped /ask output into a JSON parser 5\nseconds later and got json.decoder.JSONDecodeError: Expecting value: line 1 column 1 ā the\nserver wasn't up, curl returned empty, the parser choked on nothing. Poll /health in a loop;\nnever sleep a guessed number.
gemma4 + GEval): verdicts with reasoning, no OpenAI keyEXPECT_MODEL guard that turns silently-evaluated-the-wrong-model into an under-three-second loud failureThe suite judges answers. Part 4's Langfuse traces know why ā which chunks were retrieved,\nwhat the generation span cost, where latency hides. Wiring eval failures back to their traces\n(DeepEval test ID ā Langfuse trace ID) turns \"test failed\" into \"here's the retrieval that\ncaused it.\" That's the component-level debugging story, and it's where this series goes next.
\n![\"Promptfoo\"](\"https://wec.wiline.com/docs/assets/images/promptfoo-avatar-31b32c657d72dac0a7edbd77c24ec717.jpg\"){kind=avatar}
+Everything in this series was building to this. You can prove a model works\n(part 1), make its output machine-reliable\n(part 2), generate real test data\n(part 3), and observe production\n(part 4). Now we spend all four skills at once on\nthe pattern behind almost every serious LLM product: RAG ā retrieval-augmented generation.
\nWe'll build a docs assistant: a containerized HTTP service that answers customer questions\nfrom WEC's own documentation. Not a notebook ā a service, Docker-first, the shape you'd actually\ndeploy. And because this series doesn't do happy-path demos: along the way our RAG hallucinates a\nGPU price, we root-cause it to our own scraper, fix it, and pin the fix with a regression\ntest. Every command, number, and error below is from a real run.
\n\nOne architecture decision up front: we split the work. Embeddings run locally (a small ONNX\nmodel ā fast, free, no GPU needed) and generation runs on WEC. This is a completely standard\nproduction split: retrieval is cheap and latency-sensitive, so keeping it next to your vector\nstore saves a network round-trip per query ā while generation is where the big model earns its\nkeep.
\nThe WEC catalog now ships bge-m3 (1024-dim, multilingual, batch input) on\n/v1/embeddings:
curl -s https://inference.wiline.com/v1/embeddings \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"bge-m3\",\"input\":[\"first chunk\",\"second chunk\"]}' | jq '.data | length'
Swapping this tutorial over is a two-function change ā replace the fastembed calls in\ningest.py and main.py with the same OpenAI client pointed at /v1/embeddings\n(wec.embeddings.create(model=\"bge-m3\", input=chunks)), and re-ingest (the vector dimension\nchanges from 384 to 1024, so drop and rebuild the collection). We keep the local path below\nbecause it's free, offline, and every number in this guide was measured with it ā but both are\nproduction-legitimate; pick per your latency and ops preferences.
df -h /. Our box was at 99% and survived, but\ntight disk is the #1 silent killer of Docker builds.Real RAG features ship as services, so we start as one: a FastAPI app in a container, code\nbind-mounted so uvicorn --reload picks up every edit ā no rebuild per change.
mkdir -p ~/rag-service/app && cd ~/rag-service
cat > requirements.txt <<'EOF'
fastapi==0.115.6
uvicorn==0.34.0
chromadb==0.5.23
fastembed==0.4.2
openai==1.59.7
langfuse
requests==2.32.3
beautifulsoup4==4.12.3
EOF
cat > Dockerfile <<'EOF'
FROM python:3.12-slim
WORKDIR /srv
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ app/
EXPOSE 8000
CMD [\"uvicorn\", \"app.main:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\", \"--reload\"]
EOF
cat > docker-compose.yml <<'EOF'
services:
rag-api:
build: .
ports:
- \"8000:8000\"
volumes:
- ./app:/srv/app # live code - edit on host, uvicorn reloads
- chroma-data:/data # persisted vector index
- model-cache:/root/.cache # embedding model cache (survives rebuilds)
environment:
ANONYMIZED_TELEMETRY: \"False\"
WEC_API_KEY: ${WEC_API_KEY}
LANGFUSE_PUBLIC_KEY: ${LANGFUSE_PUBLIC_KEY}
LANGFUSE_SECRET_KEY: ${LANGFUSE_SECRET_KEY}
LANGFUSE_HOST: ${LANGFUSE_HOST}
volumes:
chroma-data:
model-cache:
EOF
Secrets live in an .env file next to the compose (compose loads it automatically ā keys never\nenter the YAML). Keep it out of version control from the start:
cat > .env <<'EOF'
WEC_API_KEY=sk-your-wec-key
LANGFUSE_PUBLIC_KEY=pk-lf-ā¦
LANGFUSE_SECRET_KEY=sk-lf-ā¦
LANGFUSE_HOST=http://<your-langfuse-host>:3000
EOF
echo \".env\" >> .gitignore
A minimal app so the container has something to serve:
\ncat > app/main.py <<'EOF'
from fastapi import FastAPI
app = FastAPI(title=\"WEC docs RAG\")
@app.get(\"/health\")
def health():
return {\"status\": \"ok\"}
EOF
docker compose up -d --build
curl -s localhost:8000/health
{\"status\":\"ok\"}
![\"The](\"https://wec.wiline.com/docs/assets/images/rag-deploy-health-850c74e1ed3eff7a20368db371456f79.png\")
\nFigure 1. One command each: the container up, /health answering, and the disk check\n(ours reads 99% ā this box lives dangerously). The build takes ~90s and a ~700 MB image.
In the real world, the knowledge base you'll want your assistant to answer from usually isn't a\ntidy folder of markdown ā it's a live website: product docs, a wiki, a help center. So the\ncorpus-acquisition skill worth learning is real scraping, not file loading.
\nFor this tutorial we scrape WEC's own documentation site. It's built with Docusaurus, and\nDocusaurus (like most site generators) publishes a sitemap.xml ā which means you can discover\nevery page programmatically instead of guessing URLs. The same approach works on any site that\nships a sitemap:
curl -s https://wec.wiline.com/docs/sitemap.xml | grep -c \"<loc>\" # 109 URLs on the whole site
We filter to the product docs (/docs/cloud_portal/), pull each page, and keep only the\n<article> element ā Docusaurus wraps the actual content in it, so nav/sidebar/footer never reach\nthe index:
cat > app/ingest.py <<'EOF'
\"\"\"Scrape the WEC docs site -> chunk -> embed locally -> Chroma.\"\"\"
import re, requests, chromadb
from bs4 import BeautifulSoup
from fastembed import TextEmbedding
SITEMAP = \"https://wec.wiline.com/docs/sitemap.xml\"
FILTER = \"/docs/cloud_portal/\" # product docs only
CHUNK, OVERLAP = 700, 100 # chars
def discover():
xml = requests.get(SITEMAP, timeout=30).text
urls = re.findall(r\"<loc>([^<]+)</loc>\", xml)
return [u for u in urls if FILTER in u]
def scrape(url):
html = requests.get(url, timeout=30).text
art = BeautifulSoup(html, \"html.parser\").find(\"article\") # Docusaurus main content
return art.get_text(\"\\n\", strip=True) if art else \"\"
def chunk(text):
out, i = [], 0
while i < len(text):
out.append(text[i:i + CHUNK])
i += CHUNK - OVERLAP
return out
if __name__ == \"__main__\":
urls = discover()
print(f\"sitemap -> {len(urls)} product-doc pages\")
docs, metas = [], []
for u in urls:
for c in chunk(scrape(u)):
docs.append(c)
metas.append({\"url\": u})
print(f\"scraped -> {len(docs)} chunks\")
embedder = TextEmbedding(\"BAAI/bge-small-en-v1.5\") # local, ~66MB ONNX, no GPU needed
vectors = [v.tolist() for v in embedder.embed(docs)]
print(f\"embedded -> {len(vectors)} vectors (dim {len(vectors[0])})\")
db = chromadb.PersistentClient(path=\"/data\")
col = db.get_or_create_collection(\"wec-docs\")
col.add(ids=[str(i) for i in range(len(docs))],
documents=docs, metadatas=metas, embeddings=vectors)
print(f\"indexed -> collection 'wec-docs' now has {col.count()} chunks\")
EOF
docker compose exec rag-api python -m app.ingest
sitemap -> 37 product-doc pages
scraped -> 281 chunks
model_optimized.onnx: 100%|āāāāāāāāāāāā| 66.5M/66.5M [00:03<00:00, 16.9MB/s]
embedded -> 281 vectors (dim 384)
indexed -> collection 'wec-docs' now has 281 chunks
![\"Ingest](\"https://wec.wiline.com/docs/assets/images/rag-ingest-2a1ffde1c98f1968bbf60532fa0a0a00.png\")
\nFigure 2. The whole product-doc corpus indexed in under a minute. The embedding model caches in\nthe model-cache volume, so it downloads exactly once.
You can turn any documentation website into a queryable vector index ā sitemap discovery, content\nextraction, chunking, local embeddings.
get_text(\"\\n\", ā¦)Our first version used get_text(\" \") ā a single space as the separator. It looked harmless and\ncaused a genuine hallucination you'll see in Step 4. Keep the newline; we'll come back to this.\n(Want to watch the bug happen on your own box? Change \"\\n\" back to \" \" in scrape(), re-run\nthe ingest, and ask the Step-4 question ā then put the newline back and re-ingest.)
You may see Failed to send telemetry event ClientStartEvent: capture() takes 1 positional argumentā¦ ā a known chromadb/posthog version clash. It's harmless (your data is fine), and the\nANONYMIZED_TELEMETRY: \"False\" env in the compose silences it.
/ask endpoint: retrieve, generate, traceāNow the service itself. Three design choices worth stating:
\nsources is deterministic.langfuse.openai (part 4's drop-in), and each stage gets an\n@observe span ā so every request produces a ask ā retrieve ā generate tree in Langfuse.timeout=60 on the client. Part 4's lesson: no default timeout means a stalled backend hangs\nyour service silently.cat > app/main.py <<'EOF'
import os, chromadb
from fastapi import FastAPI
from pydantic import BaseModel
from fastembed import TextEmbedding
from langfuse import observe, get_client
from langfuse.openai import openai # auto-traces the WEC call
app = FastAPI(title=\"WEC docs RAG\")
db = chromadb.PersistentClient(path=\"/data\")
col = db.get_or_create_collection(\"wec-docs\")
embedder = TextEmbedding(\"BAAI/bge-small-en-v1.5\")
wec = openai.OpenAI(
base_url=\"https://inference.wiline.com/v1\",
api_key=os.environ[\"WEC_API_KEY\"],
timeout=60,
)
class Ask(BaseModel):
question: str
@observe() # span: retrieve
def retrieve(question: str, k: int = 4):
qv = list(embedder.embed([question]))[0].tolist()
res = col.query(query_embeddings=[qv], n_results=k)
chunks = res[\"documents\"][0]
sources = sorted({m[\"url\"] for m in res[\"metadatas\"][0]})
return chunks, sources
@observe() # span: generate (WEC call nested inside)
def generate(question: str, chunks: list[str]) -> str:
context = \"\\n---\\n\".join(chunks)
resp = wec.chat.completions.create(
model=\"Qwen2.5-3B-Instruct\", # small on purpose: cheap + fast per query,
# and its failures teach (see Step 5)
name=\"rag-generate\",
messages=[{\"role\": \"user\", \"content\":
\"Answer the question using ONLY the context below. \"
\"If the context doesn't contain the answer, say so. \"
\"When quoting a price or number, copy it VERBATIM with its unit and the item \"
\"it belongs to; never combine numbers from different lines and never compute \"
\"new numbers from examples.\\n\\n\"
f\"Context:\\n{context}\\n\\nQuestion: {question}\"}],
)
return resp.choices[0].message.content
@app.post(\"/ask\")
@observe() # root span: the whole request
def ask(body: Ask):
chunks, sources = retrieve(body.question)
answer = generate(body.question, chunks)
get_client().flush()
return {\"answer\": answer, \"sources\": sources}
@app.get(\"/health\")
def health():
return {\"status\": \"ok\"}
EOF
docker compose up -d --build
Ask it a real customer question:
\ncurl -s localhost:8000/ask -H \"Content-Type: application/json\" \\
-d '{\"question\": \"How do I create an API key for the Inference API?\"}' | python3 -m json.tool
{
\"answer\": \"To create an API key for the Inference API, follow these steps:\\n1. Log in to the WiLine Edge Cloud.\\n2. Expand the \\\"Inference\\\" section in the left sidebar.\\n3. Click on the \\\"API Keys\\\" tab.\\n4. Click the \\\"+ Create Key\\\" button located above the keys table. ...\",
\"sources\": [
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/api/wiline-edge-cloud-inference-api/\",
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/api_keys/\",
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/examples/\",
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/models_hub/\",
\"https://wec.wiline.com/docs/cloud_portal/platform/inference/overview/\"
]
}
![\"First](\"https://wec.wiline.com/docs/assets/images/rag-first-ask-301b6dfc7a64a463e0eb1e6d93b53837.png\")
\nFigure 3. First try: the exact API-key flow from the docs, with the right page cited. Scrape ā\nembed ā retrieve ā generate, end to end.
Open Langfuse ā Tracing and you'll find the request as a span tree ā ask ā retrieve ā generate ā rag-generate ā with per-step latency and tokens. When an answer is bad, this is how you\ntell bad retrieval from bad generation (part 4's skill, now on a real app).
![\"The](\"https://wec.wiline.com/docs/assets/images/rag-trace-tree-769e5224814989dedc4896f9e4e1873f.png\")
\nFigure 4. Every /ask becomes a tree: retrieval and generation each have an address.
You have a traced, containerized RAG service answering from live-scraped documentation.
A good RAG must refuse what the docs don't cover. Probe it:
\ncurl -s localhost:8000/ask -H \"Content-Type: application/json\" \\
-d '{\"question\": \"Does WEC offer a free GPU tier for students?\"}' | python3 -m json.tool
Our first version (the get_text(\" \") scraper) answered:
\"No, WEC does not offer a free GPU tier specifically for students. The pricing for
GPU hourly is listed as $0.66/GPU-hr with a 25% off annual prepay option at
$0.10/GB-mo for block storage. There is no mention of a free GPU tier in the
provided information.\"
The refusal is right. But read the pricing sentence again: \"$0.66/GPU-hr with a 25% off annual\nprepay option at $0.10/GB-mo for block storage\" ā that's three different facts welded into one\nfalse statement, gluing a storage rate onto the GPU prepay option. Check the actual pricing page\nand you find they're simply three adjacent items:
\nThe model stitched a storage price onto \"GPU\". Why? Look at the chunk we fed it. Our\nspace-separated scrape had flattened the pricing cards into one undifferentiated line:
\n... $1.44 /mo Starting at $0.66 /GPU-hr GPU hourly 25 % off Annual prepay $0.10 /GB-mo Block storage ...
The hallucination was manufactured by our own ingestion. The model conflated prices because we\ndestroyed the structure before embedding it. This is the most classic RAG failure there is ā\ngarbage in, garbage out ā and you only catch it by probing with questions and reading the\nretrieved chunks.
\n![\"The](\"https://wec.wiline.com/docs/assets/images/rag-hallucination-18e9fb70cb217a347f31057d36b774e8.png\")
\nFigure 5. The bug in the wild: correct refusal, garbled pricing ā a storage rate glued onto the\nGPU prepay option. The sources look perfectly plausible. This is why \"it cites sources\" is not the\nsame as \"it's grounded.\"
Two fixes, both already in the code above:
\nget_text(\"\\n\", strip=True) keeps line boundaries, so price\ncards stay separate lines instead of one soup.After re-ingesting with the fix, the storage-price conflation is gone. Residual ambiguity remains ā\nthe pricing strip still contains fragments like \"$1.44/mo\" (a compute-instance price) near the GPU\nline, and truly fixing that needs table-aware extraction (see What's next). Which is exactly why\nthe next step exists: pin the fixed bug so it can never quietly return.
\nThis step took four rounds of fixes to get an honest number ā and the journey from 39% to 87%\nis the best lesson in the series, because almost nothing we fixed along the way was the RAG itself.
\nWe reuse part 3's skill ā generate a QA dataset from the corpus itself ā and grade the live\nHTTP endpoint with Promptfoo. The first generator version asked for one question per page with a\nkey_fact the answer must contain, and got 37/37 pairs back. Some labels already smelled ā a vague\n\"All paid\", menu paths instead of answers ā but let's run it naively first and let the failures\nteach us. Convert to a Promptfoo dataset with plain substring assertions (icontains: + the key\nfact) and point it at the running service. The provider is HTTP ā we grade the real API,\nexactly what a CI gate would do:
mkdir -p eval
{ echo 'question,__expected'; jq -r '[.question, (\"icontains:\" + .key_fact)] | @csv' app/qa.jsonl; } > eval/tests.csv
cat > eval/promptfooconfig.yaml <<'EOF'
description: \"RAG service eval ā QA set generated from the scraped docs + regression\"
providers:
- id: https
config:
url: http://localhost:8000/ask
method: POST
headers:
Content-Type: application/json
body:
question: \"{{question}}\"
transformResponse: json.answer
tests:
- file://tests.csv
# regression: the storage-price-as-GPU-price conflation from Step 4
- vars:
question: \"What is the pricing model for GPU hourly usage in Edge Cloud?\"
assert:
- type: not-icontains
value: \"0.10\"
- type: not-icontains
value: \"GB-mo\"
EOF
cd eval && npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
Results:
ā 15 passed (39.47%)
ā 23 failed (60.53%)
Duration: 40s (concurrency: 4)
That 39.5% is what we measured during development, while the Step-4 bug was still live. Re-run\nthe same naive eval against the finished service and it lands much higher ā 27/38:
\n![\"The](\"https://wec.wiline.com/docs/assets/images/rag-eval-icontains-284e4ea4fa5a8f2dec99a5079c84c4f7.png\")
\nFigure 6. The same naive eval against the finished service: 71.05%. Better service, same\nnoisy labels ā the 30-point gap between these two runs is all service quality, and both numbers\nmean little until you read the rows.
Read the failures before you panic. From the development run's 23 failures, most are not RAG\nfailures:
\n| Failure | What actually happened |
|---|---|
| \"Balance due\" answer: \"the amount that remains dueā¦ $0.00\" | Correct answer; the label demanded the phrase All paid ā bad label |
| Payment Information: gives the exact right menu path | Correct; label phrasing doesn't substring-match ā brittle assertion |
| GPU pricing: \"$0.10 per GB-month\" | Real failure ā the Step-4 conflation, reproducing systematically |
| TTS default format: \"not specified in the context\" | Real retrieval miss ā the chunk with the default wasn't retrieved |
Substring matching (icontains) was perfect for part 2's single JSON field and is too brittle\nfor long-form RAG answers. The obvious upgrade is part 4's skill: an LLM judge grading\nsemantic consistency (llm-rubric: assertions + a judge model in defaultTest). We did exactly\nthat ā same 37 labels, judge instead of substrings ā and got:
Results:
ā 17 passed (44.74%)
ā 21 failed (55.26%)
Duration: 8m 21s (concurrency: 4)
39.5% ā 44.7%. Barely moved ā because the judge faithfully enforces bad labels. Rows with\ngarbage key_facts still fail: the reference is wrong, not the answer. A better grader cannot\nrescue a bad dataset. The bottleneck was never the assertion type.
So fix the data (part 3's discipline, now enforced by code). Generator v2 demands the label be\nthe answer ā not a menu path, not a section name ā and adds a curation gate: the key_fact\nmust exist verbatim in the source text, or the row is rejected on the spot:
cat > app/gen_qa.py <<'EOF'
\"\"\"Generate a QA eval set from the indexed docs ā v2, curated labels.\"\"\"
import os, json, chromadb
from openai import OpenAI
wec = OpenAI(base_url=\"https://inference.wiline.com/v1\",
api_key=os.environ[\"WEC_API_KEY\"], timeout=60)
db = chromadb.PersistentClient(path=\"/data\")
col = db.get_or_create_collection(\"wec-docs\")
data = col.get(include=[\"documents\", \"metadatas\"])
pages = {}
for doc, meta in zip(data[\"documents\"], data[\"metadatas\"]):
pages.setdefault(meta[\"url\"], []).append(doc)
kept, skipped = 0, 0
with open(\"/srv/app/qa.jsonl\", \"w\") as f:
for url, chunks in sorted(pages.items()):
ctx = \"\\n\".join(chunks[:2])[:2000]
resp = wec.chat.completions.create(
model=\"Qwen2.5-3B-Instruct\", temperature=0.3,
messages=[{\"role\": \"user\", \"content\":
\"From this documentation excerpt, write ONE question a customer would ask, \"
\"and the fact that ANSWERS it. Return ONLY JSON with keys:\\n\"
\"question (string), key_fact (string).\\n\"
\"Rules for key_fact: it must be THE ANSWER to the question (not a menu path, \"
\"not a section name), a short phrase copied VERBATIM from the excerpt, \"
\"max 8 words.\\n\\n\"
f\"Excerpt:\\n{ctx}\"}],
)
raw = resp.choices[0].message.content
try:
start, end = raw.index(\"{\"), raw.rindex(\"}\") + 1
row = json.loads(raw[start:end])
# curation gate: the label must literally exist in the source text
if row[\"key_fact\"].lower().strip() not in ctx.lower():
print(f\"SKIP {url.split('/docs/')[1]}: key_fact not verbatim in source\")
skipped += 1
continue
row[\"url\"] = url
f.write(json.dumps(row) + \"\\n\")
kept += 1
print(f\"ok {url.split('/docs/')[1]}: {row['question'][:50]} -> {row['key_fact'][:40]}\")
except Exception as e:
print(f\"SKIP {url}: {e}\")
skipped += 1
print(f\"done -> {kept} kept, {skipped} rejected by curation gate\")
EOF
docker compose exec rag-api python -m app.gen_qa
ok cloud_portal/management/billing/how_billing_works/: What is the starting point ... -> How Billing Works
SKIP cloud_portal/management/billing/overview/: key_fact not verbatim in source
ok cloud_portal/platform/inference/api/wiline-edge-cloud-inference-api/: What is the base URL ... -> https://inference.wiline.com/v1
SKIP cloud_portal/platform/inference/api_keys/: key_fact not verbatim in source
...
done -> 15 kept, 22 rejected by curation gate
15 kept, 22 rejected. That rejection rate is the point: more than half of what the generator\nproduced would have graded the RAG against wrong references. A smaller, trustworthy dataset beats\na bigger, poisoned one.
\nThe generator samples at temperature 0.3, so the exact tally shifts slightly between runs ā here's\na later regeneration that kept 16 of 37; the rejection rate stays brutal either way:
\n![\"The](\"https://wec.wiline.com/docs/assets/images/rag-qa-gen-8dfb631c47f8fc4cb9c506fef0f651d2.png\")
\nFigure 7. The gate at work: 16 kept, 21 rejected on this run. Every surviving label is provably\nin the docs ā real answer-facts like audio/mpeg and the API base URL, not menu paths. (The\ndocumented eval runs below use our first curated set: 15 pairs + the regression = 16 tests.)
Rebuild tests.csv with a rubric that gives the judge the question and the verified\nreference, plus explicit PASS/FAIL rules (different wording is fine; contradiction fails):
{ echo 'question,__expected'; jq -r '[.question, (\"llm-rubric:Question asked: \\\"\" + .question + \"\\\". Reference fact from the official docs: \\\"\" + .key_fact + \"\\\". PASS if the answer correctly addresses the question and does not contradict the reference fact ā different wording is fine. FAIL only if the answer is wrong, contradicts the reference, or fails to answer the question.\")] | @csv' ../app/qa.jsonl; } > tests.csv
Results:
ā 12 passed (75.00%)
ā 4 failed (25.00%)
Duration: 2m 47s (concurrency: 4)
75%. Better ā but before celebrating, read the four failures. And here the series' habit pays\noff one more time, because three of them weren't RAG failures either.
\nThe three failures whose grader reason was just \"No output\" all shared two tells:\ngraderError: true, and a completion of exactly 1024 tokens ā Promptfoo's default\nmax_tokens. Our judge is a thinking model: it spent the whole budget reasoning and got cut off\nbefore emitting the verdict JSON, which Promptfoo counts as FAIL. The fourth \"failure\" got a\nverdict ā but the small judge had literally echoed the schema placeholder (reason: \"string\")\ninstead of grading. Both answers were actually correct.
Two config lines fix it ā a bigger judge, and room to think:
\ndefaultTest:
options:
provider:
id: openai:chat:Qwen3.5-122B
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
# thinking models spend tokens on reasoning before the verdict JSON;
# the 1024 default silently truncated the judge and failed 3 tests
max_tokens: 4096
Results:
ā 14 passed (87.50%)
ā 2 failed (12.50%)
And the row that matters most:
\nā What is the pricing model for GPU hourly usage in Edge Cloud? ā [PASS] ... starting at $0.66/GPU-hour. ā
The regression is green ā the Step-4 conflation is pinned and cannot quietly return. (One\ncosmetic note: the 122B judge sometimes keeps its reason text to a terse ... ā the verdicts\nthemselves are correct, which is what the gate reads.)
You can gate a live RAG service in CI: generated + gate-curated QA, a judge for semantics,\ndeterministic regressions for known bugs ā and you know how to debug the judge itself.
One habit this series drills: re-run before you believe. We did ā and got 75%, with\ndifferent rows failing than before. Billing, which had just passed, now failed with a circular\necho; the TTS answer said \"mp3\" one run and \"unspecified\" the next. The score was a lottery\nbecause the service itself was non-deterministic: generate() passed no temperature, so\nevery eval graded a different roll of the dice at the default 1.0.
One line ends the lottery ā add to the create() call in generate():
temperature=0, # determinism: same question -> same answer
Re-run twice. 13/16 ā 81.25% ā both times, with the same three failures. The number went\ndown and that's the win: 87.5% was partly luck (the flaky rows happened to land right that run);\n81.25% is reproducible. A reader who re-runs your eval should get your number.
\nAfter four rounds of fixing the measurement, what's left is signal ā stable across runs, and\neach one is a different classic RAG failure type:
\n\"iated with subnets\\nData Transferā¦\"). Fixed-size character chunking\nsplit the fact across a boundary, and the half with the answer doesn't rank. Fix class: larger\noverlap, bigger k, or structure-aware chunking.A trustworthy eval isn't the goal; it's the map. It just told us two of three failures are the\nsame class ā facts that don't reach the top-4 retrieved chunks. The cheapest fix in that class is\none character, in retrieve():
def retrieve(question: str, k: int = 6): # was 4 ā two misses said \"not enough context\"
Re-run. Then re-run again, because that's the habit now:
\nResults:
ā 14 passed (87.50%)
ā 2 failed (12.50%)
14/16 ā 87.5% ā twice, same two failures. Read what happened with each:
\nk=6 the chunk with the stats block doesn't rank: this miss is\nsemantic, not budgetary ā the boundary-mangled text (\"iated with subnetsā¦\") embeds poorly.\nMore k can't fix what chunking broke; this one genuinely needs structure-aware chunking.The number is back to 87.5% ā the same as the lucky run ā but this time it reproduces, and\nevery point of it has an explanation.
\n![\"Final](\"https://wec.wiline.com/docs/assets/images/rag-eval-final-eb3776d3a5abed5af34d6896bbee06d0.png\")
\nFigure 8. The final run: 14/16, the same two failures every time, regression PASS.
39.5% ā 44.7% ā 75% ā 87.5% (lucky) ā 81.25% (honest) ā 87.5% (earned). Until the very\nlast step, the RAG barely changed ā what changed was the measurement: assertions (substrings ā\njudge), labels (curation gate), the judge's own config (truncation), determinism (temperature).\nOnly then did one retrieval parameter, chosen by the eval, actually move quality. That's the\nlast lesson of the series:
\nPOST /ask with\ndeterministic citations.Parts 1ā4 were the toolkit. This is what the toolkit is for.
\n/ask returns nothing after re-indexingāIf you drop and recreate the Chroma collection while the API is running, the service holds a\nstale collection handle and every query throws:
\nchromadb.errors.InvalidCollectionException: Collection 9b486740-ā¦ does not exist.
The running process cached the old collection's UUID. Restart the service\n(docker compose restart rag-api) ā or resolve the collection per-request instead of at startup.
![\"The](\"https://wec.wiline.com/docs/assets/images/rag-stale-handle-1639bfb9ef1938577ff551daa191a889.png\")
\nFigure 9. Re-indexing invalidates open handles ā a classic \"works until you rebuild the index\"\nproduction trap.
Failed to send telemetry event ā¦ capture() takes 1 positional argumentāHarmless chromadb/posthog version clash. Set ANONYMIZED_TELEMETRY: \"False\" in the environment\n(already in our compose).
graderError: true)āCheck the grading record's completion tokens: if it's exactly 1024 ā Promptfoo's default\nmax_tokens ā your thinking-model judge spent the whole budget reasoning and was cut off before\nthe verdict JSON. Set max_tokens: 4096 on the judge provider. Related tell: a small judge\nreturning reason: \"string\" is echoing the schema placeholder instead of grading ā use a bigger\njudge.
Reasoning judges are slow (~minutes for a dozen rows) and imperfect ā expect occasional verdict\nflips between runs. Keep deterministic assertions for known bugs (they're free and never flip)\nand treat the judge score as a trend, not a truth. If a judge row hangs entirely, it's usually a\ntransient network hiccup, not your config ā retry before debugging.
\nThe image needs ~1 GB (chromadb + onnxruntime are the heavy layers). Check df -h / first and\ndocker system df for reclaimable space ā but never prune blindly on a shared box: \"reclaimable\"\nlies when other projects' containers are running.
The honest limitations are the roadmap, and each one traces to a failure you watched happen:\nstructure-aware chunking (the subnets miss survived k=6 ā the boundary-mangled chunk embeds\npoorly, so no retrieval budget rescues it), prompt or model work on generation (the billing\necho: force the model to name the page it's citing, or use a stronger generator), table-aware\nextraction (the pricing strip is still fragment soup ā the regression passes, but \"$1.44/mo\"\nstill bleeds near the GPU line; add a positive assert for the correct figure, not just\nnot-icontains for the wrong ones), and judge calibration (grade the judge against a handful\nof human verdicts). Each one moves the pass rate for a reason you can name ā which, after five\nparts, is the whole point: you don't guess whether your AI works. You measure it, watch it, and\nmake it prove itself.
A customer says your support bot promised them a refund policy that doesn't exist. Your feature\nmade two LLM calls ā classify, then reply. Which one invented it? If you can't answer that,\nyour app is a black box ā and this guide fixes exactly that.
\nYou can now prove a model works (part 1),\nmake its output machine-reliable (part 2),\nand generate a real test set to check it against\n(part 3). But all of that runs offline,\nin CI, on inputs you chose. Production doesn't play along.
\nThis guide closes the gap. We'll self-host Langfuse ā the open-source, self-hostable\nalternative to LangSmith ā trace every real call, auto-score live traffic with an LLM judge,\ndrill into the exact step that breaks, and feed failures back so your part-3 dataset gets\nstronger. Offline eval tells you it worked on your test set; this tells you it works in the wild.
\n\nBy the end you can take any WEC feature and: see inside every call, score quality on real\ntraffic automatically, find the exact call that broke and why, and turn production failures into\nnew test cases. That's the difference between \"it passed CI\" and \"it's holding up in production.\"
\nWe'll keep the running example from part 3 ā a support-ticket classifier on the WEC Inference\nAPI ā and put it under observation.
\nYour app calls the WEC API as usual; Langfuse (self-hosted) quietly captures every call as a\ntrace, an LLM judge scores each one, and the failures flow back into your part-3\ndataset ā so production keeps making your evals stronger:
\n\nWe use Langfuse because it's the open-source leader for LLM observability (MIT-licensed,\nself-hostable), its SDK is OpenAI-compatible so it points at the WEC API unchanged, and ā the\nreason it fits this series ā it has datasets, scoring, and LLM-as-judge evaluators built in.\nThat's what lets your part-3 dataset and the \"score production, feed failures back\" loop be\nfirst-class instead of bolted on.
\nThe trade-off: the production stack is heavy (Postgres + ClickHouse). If you don't need the eval\nfeatures and want lighter, reasonable alternatives are Arize Phoenix (OpenTelemetry-native,\nstrong for offline eval), Helicone (a one-line proxy ā but no evals), or OpenLIT\n(single-binary, OTEL-native). For an eval-driven workflow like this one, the extra containers earn\ntheir keep; for plain tracing, one of those may suit you better.
\nLangfuse's self-host stack is six services (web + worker, ClickHouse, Postgres, Redis, object\nstore), so give it a WEC Instance with ~15 GB disk and 4 GB+ RAM ā don't cram it onto a box\nthat's already busy.
\ngit clone https://github.com/langfuse/langfuse.git && cd langfuse
Generate three secrets and drop them in a .env (the ENCRYPTION_KEY must be hex, or the\nweb container won't boot):
cat > .env <<EOF
NEXTAUTH_SECRET=$(openssl rand -base64 32)
SALT=$(openssl rand -base64 32)
ENCRYPTION_KEY=$(openssl rand -hex 32)
NEXTAUTH_URL=http://<your-instance-ip>:3000
EOF
docker compose up -d
docker compose ps
![\"The](\"https://wec.wiline.com/docs/assets/images/langfuse-deploy-ps-597bc541cd07b60e848e6c0dfe6246fc.png\")
\nFigure 1. docker compose ps ā all six services (web, worker, ClickHouse, Postgres, Redis, MinIO) up and healthy.
If the box already runs something on port 3000 or 5432, Langfuse won't start (\"address already\nin use\"). The backing databases don't need host ports at all ā edit docker-compose.yml to move\nlangfuse-web to a free host port (e.g. 3001:3000) and drop the Postgres host mapping. Keeping\nyour databases off the host network is good practice anyway. See Troubleshooting.
Open http://<instance-ip>:3000, create an account (first user is the owner), then an\nOrganization ā Project, and under Settings ā API Keys create a key pair\n(pk-lf-ā¦ / sk-lf-ā¦).
Point the Langfuse SDK at your instance and use its OpenAI drop-in ā it auto-traces any\nOpenAI-compatible call, including the WEC Inference API, with zero extra code around each request:
\npython3 -m venv ~/lf-env && source ~/lf-env/bin/activate
pip install langfuse openai
export LANGFUSE_PUBLIC_KEY=\"pk-lf-ā¦\"
export LANGFUSE_SECRET_KEY=\"sk-lf-ā¦\"
export LANGFUSE_HOST=\"http://<instance-ip>:3000\" # use the port you exposed ā 3001 if you remapped in Step 1
export WEC_API_KEY=\"sk-your-wec-key\"
import os
from langfuse.openai import openai # drop-in: auto-traces every call
from langfuse import get_client
client = openai.OpenAI(
base_url=\"https://inference.wiline.com/v1\",
api_key=os.environ[\"WEC_API_KEY\"],
timeout=30, # ALWAYS set this ā a slow backend hangs forever otherwise
)
def classify(ticket: str) -> str:
resp = client.chat.completions.create(
model=\"Qwen2.5-3B-Instruct\",
name=\"ticket-classify\", # names the trace in Langfuse
messages=[{\"role\": \"user\", \"content\":
\"Classify this support ticket. Return ONLY JSON with keys \"
\"category (billing/technical/account/other), priority (low/medium/high), \"
f\"summary. Ticket: {ticket}\"}],
)
return resp.choices[0].message.content
if __name__ == \"__main__\":
print(classify(\"I was charged twice this month.\"))
get_client().flush() # push traces before the process exits
Run it, then open Tracing ā the call shows up in seconds with input, output, latency, token\ncounts, and (once we add pricing) cost.
\n![\"A](\"https://wec.wiline.com/docs/assets/images/langfuse-first-trace-f013e891fddc5d65b3525d5e1ad59fb7.png\")
\nFigure 2. One ticket-classify call in Langfuse: the prompt, the JSON output, latency (~1.2s),\nand token counts ā captured with zero code wrapped around the request.
The OpenAI client has no default timeout. If the model is slow or the backend stalls, your\napp hangs indefinitely ā no error, no trace. timeout=30 turns that into a fast, visible failure.
One call is easy to log; you don't need Langfuse for that. Tracing earns its keep on multi-step\nfeatures, where it shows the whole chain as one tree so you can see which step broke. Let's make\nthe feature realistic ā classify the ticket, then draft a reply from that category (two WEC calls) ā\nand group them into a single trace with the @observe decorator:
from langfuse import observe, get_client
from app import client, classify # reuse the traced WEC client + classifier
def draft_reply(ticket: str, category: str) -> str:
resp = client.chat.completions.create(
model=\"Qwen2.5-3B-Instruct\",
name=\"draft-reply\",
messages=[{\"role\": \"user\", \"content\":
f\"A customer sent this {category} ticket. Write a short, helpful reply.\\nTicket: {ticket}\"}],
)
return resp.choices[0].message.content
@observe() # groups everything below into ONE trace
def handle_ticket(ticket: str) -> str:
category = classify(ticket) # child span 1
reply = draft_reply(ticket, category) # child span 2
return reply
if __name__ == \"__main__\":
print(handle_ticket(\"I was charged twice this month.\"))
get_client().flush()
Open that trace and you get a tree: handle_ticket ā ticket-classify ā draft-reply, each\nchild with its own input, output, latency, and tokens. Now a failure has an address ā you can\npoint at the exact step that broke instead of guessing (we use this in Step 5).
You can now see inside any LLM call in production ā input, output, latency, tokens ā without\nwriting a line of logging code.
To get real signal (not one call), push your part-3 dataset through the classifier so the\ndashboard has traffic to analyze:
\nimport os, json
from langfuse.openai import openai
from langfuse import get_client
from app import classify # reuse the traced function
for line in open(os.path.expanduser(\"~/tickets_dataset.jsonl\")):
if line.strip():
classify(json.loads(line)[\"ticket\"])
get_client().flush()
Now the Home dashboard shows traces over time, a latency distribution (p50/p95), tokens by\nmodel, and a cost total ā the \"watch it in production\" view.
\n![\"The](\"https://wec.wiline.com/docs/assets/images/langfuse-scores-dashboard-e46938c8695864fa98dc8afa0e488b3a.png\")
\nFigure 3. After replaying the part-3 dataset: traces over time, a latency distribution, tokens\nby model, and a cost total ā the \"watch it in production\" view.
Langfuse auto-computes cost for models it knows (OpenAI, Anthropic), but the WEC Inference API\nreturns no dollar cost and Langfuse doesn't know WEC's model prices ā so cost shows $0. This is\nthe same gap we hit in part 1. Fix it once:\nSettings ā Models ā add a model matching Qwen2.5-3B-Instruct with your per-token input/output\nprices. Now every trace and the dashboard show real spend ā the cost-tracking that Promptfoo\ncouldn't give us.
CI eval checks a fixed set. Here we score every real call automatically with an LLM-as-judge.\nThis step looks like pure configuration ā it's also where we hit, and solved, the best bug of the\nseries.
\nCreate the evaluator via Evaluators ā + New evaluator. It's a 3-step wizard:
\n1. Select Evaluator ā pick the Correctness LLM-as-judge template.
\n2. Set up LLM connection ā the judge needs a model to call, and there's none yet, so click\n+ Add LLM Connection. The dialog has more fields than you'd expect ā here's exactly what each\none needs:
\nopenaiwec-judge) ā just the connection's name in Langfusehttps://inference.wiline.com/v1 ā critical; left blank it hits real OpenAIQwen3.5:9B ā the model choice matters a lot\nhere, as you're about to seewec-judge and model Qwen3.5:9B ā Save.![\"The](\"https://wec.wiline.com/docs/assets/images/langfuse-llm-connection-e83545064c144db968575c7d85d0c809.png\")
\nFigure 4. The New LLM Connection dialog: adapter openai, the WEC API Base URL, default\nmodels switched off, and a custom model added ā this is what points the judge at WEC instead of\nreal OpenAI.
3. Run Evaluator ā add a filter Name = ticket-classify, toggle Run on live incoming\nobservations so it scores new traffic, and Execute.
![\"The](\"https://wec.wiline.com/docs/assets/images/langfuse-evaluator-config-328cb01c6a061467eca223c1d7266d31.png\")
\nFigure 5. The Correctness evaluator ā score name, the ticket-classify filter, and \"run on\nlive incoming observations.\"
We configured everything, sent fresh trafficā¦ and nothing. No score on any trace. Here's how we\ntracked it down ā you may hit the same wall:
\nRequest timed out after 120000ms. Every run, exactly 120s ā so the judge was firing, its\nLLM call just never came back.Qwen2.5-3B-Instruct, rejected tool calls outright\n(instant 400 ā a missing vLLM config on the backend; more on that below). We switched to\nQwen3.5:9B, which answers a direct tool call in ~3s. Still timed out.curl from the host: 200 in half a second. The same judge-style\nrequest from inside the worker container: ~3s. Model, network, container ā all fine.response_format) so the score comes back machine-readable. Replaying\nthat exact call shape exposed the problem: on this backend, judge-style requests to the\nreasoning model are wildly slow and unpredictable ā the identical request ranged from 3 to\n95 seconds across runs (the model \"thinks\" at length, and JSON mode adds overhead). With the\njudge's longer real prompt, runs blew past the 120s default. Nothing was broken ā it was just\nslower than the timeout.The fix is two lines of configuration. Langfuse reads its LLM timeout from\nLANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS (default 120000). Give it room:
services:
langfuse-worker:
environment:
LANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS: \"600000\"
langfuse-web:
environment:
LANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS: \"600000\"
docker compose up -d langfuse-worker langfuse-web
# verify it landed inside the container:
docker compose exec langfuse-worker env | grep TIMEOUT
Send a fresh trace through app.py, wait a couple of minutes, and:
![\"The](\"https://wec.wiline.com/docs/assets/images/langfuse-score-on-trace-99653e359fac7e126dc097103602a4e5.png\")
\nFigure 6. There it is ā Correctness: 1.00 attached to a live ticket-classify trace, with\nthe judge's reasoning a click away. Every new call now gets scored automatically.
You can now auto-grade real production traffic ā and debug an evaluator that silently produces\nnothing: check its Logs, isolate model vs. network vs. container, match the timeout to reality.
The judge needs tool calling. If your model returns a 400 on any tools request, that's\nusually the serving config, not the model ā on vLLM it's the --enable-auto-tool-choice and\n--tool-call-parser flags. That was our case: we reported it, the platform team enabled the\nflags on Qwen2.5-3B-Instruct the same day, and the 400 disappeared. Only the judge's connection\nis affected ā your app keeps its model.
Scoring is slow (~2 min/score) and that's the backend, not you. The reasoning model \"thinks\"\nat length on judge prompts ā worth reporting to your platform team (reasoning budget caps, or a\nguided-decoding backend like xgrammar on vLLM). The raised timeout keeps scoring reliable meanwhile.
Once tool calling was enabled on Qwen2.5-3B, we tried it as the judge ā scores came back in\n2ā10 seconds instead of ~2 minutes. But they were garbage: the same kind of correct answer got\n1, 1, 0, and 0.05 across four runs, with near-identical rationales attached to opposite\nscores. The small model fills the score format perfectly; it just can't actually evaluate.\nMirror image of part 3's finding: the small\nmodel wins at classifying and loses at judging ā judging is a reasoning task. Keep the judge\non Qwen3.5:9B: slow but coherent.
This is where the span tree pays off. Open a handle_ticket trace and you don't get one opaque\nblob ā you get the whole chain as a tree, each step with its own input, output, latency, and\ntokens:
![\"Drilling](\"https://wec.wiline.com/docs/assets/images/langfuse-failing-trace-8fcb77412fc10f743dad85e747382695.png\")
\nFigure 7. The handle_ticket tree ā handle_ticket ā ticket-classify ā draft-reply ā with\nper-step latency and tokens. Every step has an address you can inspect.
That structure is what makes production failures debuggable. Say a customer gets a nonsense reply.\nWithout tracing you'd only know \"the reply was bad.\" With the tree you expand it and read each step\nā for example:
\nticket-classify ā category: \"billing\" ā
correct ā so the classifier isn't the problem.draft-reply ā invented a refund policy that doesn't exist ā ā the bug is in step 2.You've localized the failure to the exact step, input, and output ā you'd fix the draft-reply\nprompt, not waste time on the classifier. Offline eval tells you the pass rate dropped; the trace\ntells you which call, on which input, produced which wrong output, at which step. And now that\nStep 4 scores every call, you don't even have to hunt: filter Tracing by low score and the\nsuspicious traces surface themselves ā often a phrasing your part-3 dataset never covered, which is\nexactly what you capture back next.
You can now pin a production failure to the exact step, input, and output ā no more guessing\nwhich call invented that refund policy.
A production failure is a missing test case ā and Langfuse makes capturing it one click. Create\nthe dataset once (Datasets ā + New dataset, e.g. production-failures), then on any trace click\n+ Add to datasets and pick it. The trace's input and output are prefilled as a new item:
![\"Adding](\"https://wec.wiline.com/docs/assets/images/langfuse-dataset-loop-9e09a4627451a945e4df5245e6018fe7.png\")
\nFigure 8. \"Add to datasets\" turns a production trace into a new test case ā input and expected\noutput prefilled ā feeding it straight back into your part-3 dataset.
To automate it, pull the traces you've bookmarked or added to a dataset via the Langfuse SDK, write\n{ticket, category, priority} rows into tickets_dataset.jsonl, and re-run the part-1/part-3\nPromptfoo eval ā now it covers the real-world case.
Now your CI eval is stronger because production found a hole in it. That's the flywheel:\noffline eval ā ship ā observe & score real traffic ā capture failures ā offline eval gets\nstronger.
\nYou've closed the loop: production failures now make your evals stronger ā the flywheel that\nkeeps an AI feature good after it ships.
That's the \"keep it good after shipping\" rung. Offline eval (parts 1ā3) proves it works before\nlaunch; this proves ā and keeps ā it working after.
\ndocker compose upāAnother service holds port 3000 (web) or 5432 (Postgres). The databases don't need host ports ā\nin docker-compose.yml, remap langfuse-web to a free port (3001:3000) and remove the Postgres\nhost mapping. Compose merges ports lists, so an override file with ports: [] won't remove\nthem; edit the compose directly.
Almost always ENCRYPTION_KEY ā it must be openssl rand -hex 32 (64 hex chars), not base64.\nCheck docker compose logs langfuse-web.
$0āLangfuse has no price for WEC's models by default. Add the model + per-token prices under\nSettings ā Models (Step 3).
\nCheck Evaluators ā your evaluator ā Logs first ā an error there tells you it's running and\nfailing, not idle. Then match the error:
\ntool_choice ā tool calling isn't enabled for that model on\nthe serving backend (on vLLM: the --enable-auto-tool-choice / --tool-call-parser flags).\nTwo ways out: point the judge's LLM Connection at a model where it works (Qwen3.5:9B), or ask\nyour platform team to enable the flags ā that's what fixed Qwen2.5-3B-Instruct for us. Either\nway, only the judge changes ā your app keeps its model.Request timed out after 120000ms ā the judge's structured-output call is slower than\nLangfuse's default timeout. Raise LANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS (Step 4) and\nconfirm with docker compose exec langfuse-worker env | grep TIMEOUT. If plain curl is fast\nbut scoring still times out, don't chase the network ā the structured-output path is what's\nslow, and that's a backend fix.That's not a hang ā it's the reasoning model generating long hidden chain-of-thought on judge\nprompts (highly variable: the same request can take 3s or 95s), plus structured-output overhead.\nRaising the timeout makes scoring reliable but not fast ā the real fix is platform-side\n(reasoning budget caps, guided-decoding backend).
\nYou can now build, prove, observe, and improve a single WEC feature. The last rung is the real\napp pattern: RAG ā retrieval + generation ā with everything from parts 1ā4 baked in\n(schema-checked output, an eval dataset, and Langfuse tracing over the whole pipeline). That's\nwhere it all comes together.
\n++Your eval passes 100% ā of the two test cases you typed by hand. Real users won't phrase\nthings the way you did.
\nIn part 1 and\npart 2 we built a real eval harness ā\nassertions, schema validation, a model matrix. But two cases can't tell you if your app works;\nthey can only tell you it didn't crash on two inputs.
\nThis guide fixes that. The skill is producing a dataset you can actually trust: we use the\nWEC Inference API to generate labeled tickets, then validate and curate them ā because\ngenerated labels are not automatically correct, and treating them as gospel just moves the bug.\nThe result is real test data at scale. When you finally run it through the part-2 harness, coverage\nsurfaces the failures ā genuine misclassifications and debatable labels ā that two hand-picked\ncases hide.
\n\nEach step feeds the next, ending in a dataset that plugs straight into the part-2 eval:
\n\nA WEC Inference API key ā see Inference ā API Keys.
\njq, curl, and Node.js 22+ (Promptfoo runs via npx).
Your key exported and sanitized (a pasted key often carries invisible characters):
\nexport WEC_API_KEY='sk-your-key'
export WEC_API_KEY=$(printf '%s' \"$WEC_API_KEY\" | LC_ALL=C tr -cd '[:print:]')
If a call returns a 401 / \"invalid API key\", your key is expired or revoked ā generate a\nfresh one in Inference ā API Keys. See Troubleshooting.
Ask the model for tickets with their answers ā the labels are what make it a dataset, not\njust inputs. We reuse the exact schema from part 2 (category + priority enums):
curl -s https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{
\"model\": \"Qwen2.5-3B-Instruct\",
\"temperature\": 0.7,
\"messages\": [{\"role\":\"user\",\"content\":\"Generate 8 diverse, realistic customer support tickets for a cloud hosting company. Return ONLY JSONL (one JSON object per line), each with keys: ticket (string), category (one of: billing, technical, account, other), priority (one of: low, medium, high). No markdown, no fences.\"}]
}' | jq -r '.choices[0].message.content'
{\"ticket\":\"Our server went down unexpectedly and we're not sure what caused it.\",\"category\":\"technical\",\"priority\":\"high\"}
{\"ticket\":\"We need to upgrade our plan but the system is showing an error message when trying to change plans.\",\"category\":\"account\",\"priority\":\"medium\"}
{\"ticket\":\"We're having trouble accessing our database and can't seem to log in.\",\"category\":\"technical\",\"priority\":\"high\"}
... 5 more lines ...
![\"Step](\"https://wec.wiline.com/docs/assets/images/dataset-gen-batch-adc9c63dcd4af4c2991e7581ea5908ec.png\")
\nFigure 1. A single generation: 8 diverse tickets, each already carrying its category and priority label.
Clean JSONL, valid labels, real diversity ā and it came back in seconds. We deliberately picked a\nsmall, non-reasoning model (Qwen2.5-3B-Instruct) for generation. Here's why that matters.
Reasoning models return their chain-of-thought in a separate reasoning_content field ā tokens\nyou pay for but never use. Ask a reasoning model like Qwen3.5:9B for a few tickets and compare the\nhidden reasoning against the actual answer:
curl -s https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"Qwen3.5:9B\",\"temperature\":0.7,\"messages\":[{\"role\":\"user\",\"content\":\"Generate 3 realistic customer support tickets for a cloud hosting company. Return ONLY JSONL, keys: ticket, category, priority. No markdown, no fences.\"}]}' \\
| jq '{model, completion_tokens: .usage.completion_tokens, reasoning_chars: (.choices[0].message.reasoning_content|length), answer_chars: (.choices[0].message.content|length)}'
{
\"model\": \"Qwen3.5:9B\",
\"completion_tokens\": 2000,
\"reasoning_chars\": 7116,
\"answer_chars\": 412
}
![\"A](\"https://wec.wiline.com/docs/assets/images/dataset-gen-usage-5ee23a331ab19ce10dfddd509162743d.png\")
\nFigure 1b. For just 3 tickets, Qwen3.5:9B wrote 7,116 characters of hidden reasoning to\nproduce 412 of answer ā you pay for all ~2,000 completion tokens. (Counts vary per call; the\nlopsidedness doesn't.) That's why we generate with the small, non-reasoning Qwen2.5-3B ā and it's\nthe same reason those reasoning models stumble on structured output in Step 5's matrix.
Never treat generated data as ground truth without checking it. Save a batch and validate three\nthings: every line parses, every label is in-enum, and the classes are reasonably balanced.
\ncurl -s https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"Qwen2.5-3B-Instruct\",\"temperature\":0.7,\"messages\":[{\"role\":\"user\",\"content\":\"Generate 8 diverse, realistic customer support tickets for a cloud hosting company. Return ONLY JSONL (one JSON object per line), each with keys: ticket (string), category (one of: billing, technical, account, other), priority (one of: low, medium, high). No markdown, no fences.\"}]}' \\
| jq -r '.choices[0].message.content' > tickets.jsonl
# 1) do all lines parse as JSON?
jq -e . tickets.jsonl > /dev/null && echo \"all lines valid JSON\" || echo \"INVALID JSON present\"
# 2) any out-of-enum labels? (no rows printed = all valid)
jq -c 'select((.category|IN(\"billing\",\"technical\",\"account\",\"other\")|not) or (.priority|IN(\"low\",\"medium\",\"high\")|not))' tickets.jsonl
# 3) class distribution
jq -rs 'group_by(.category)[] | \"\\(.[0].category): \\(length)\"' tickets.jsonl
all lines valid JSON
account: 2
billing: 2
other: 1
technical: 3
![\"Step](\"https://wec.wiline.com/docs/assets/images/dataset-validate-9b1201646866c2062c5e9aa3d093cfd2.png\")
\nFigure 2. Validation output ā every line parses, no out-of-enum labels, and a reasonable class spread.
The intuitive [\"billing\",...] | index(.category) fails with \"Cannot index array with\nstring\" ā inside the pipe, . is the array, so .category tries to index it. Use\n.category | IN(\"billing\", ...) instead. See Troubleshooting.
One batch of 8 isn't enough. Loop the call a few times (higher temperature for variety across\nbatches), concatenate, then dedupe by ticket text:
\nfor i in $(seq 1 5); do
curl -s https://inference.wiline.com/v1/chat/completions \\
-H \"Authorization: Bearer $WEC_API_KEY\" -H \"Content-Type: application/json\" \\
-d '{\"model\":\"Qwen2.5-3B-Instruct\",\"temperature\":0.9,\"messages\":[{\"role\":\"user\",\"content\":\"Generate 8 diverse, realistic customer support tickets for a cloud hosting company. Return ONLY JSONL (one JSON object per line), each with keys: ticket (string), category (one of: billing, technical, account, other), priority (one of: low, medium, high). No markdown, no fences.\"}]}' \\
| jq -r '.choices[0].message.content // empty'
echo \"batch $i done\" >&2
done > tickets_raw.jsonl
# dedupe by ticket text
jq -sc 'unique_by(.ticket)[]' tickets_raw.jsonl > tickets_dataset.jsonl
echo \"raw: $(wc -l < tickets_raw.jsonl) ā deduped: $(wc -l < tickets_dataset.jsonl)\"
raw: 40 ā deduped: 40
![\"Step](\"https://wec.wiline.com/docs/assets/images/dataset-scale-dedupe-74ac55864b3da488aa334816e11ede4a.png\")
\nFigure 3. Scaling up: 5 batches ā 40 rows after exact-duplicate removal.
Zero exact duplicates across 5 batches ā a good sign the model isn't just repeating itself.
\nunique_by(.ticket) only catches identical text. Two differently-worded \"server is down\"\ntickets are semantic near-duplicates and will slip through ā catching those needs\nembedding-based clustering, a more advanced step. For a starter dataset, exact dedup is fine.
Note the // empty in the jq filter. If a call fails (revoked key, model timeout), the\nresponse has no content and .choices[0].message.content is null ā a plain jq -r would then\nwrite the literal word null into your dataset, silently poisoning it. // empty drops those,\nso a failed call adds nothing instead of a junk row. Always check the row count afterward.\nSee Troubleshooting.
Validate the full 40 the same way as Step 2 (jq -e ., the enum check, and both distributions).\nOurs came out balanced: technical 15 Ā· account 10 Ā· billing 10 Ā· other 5; priority high 13 Ā·\nmedium 14 Ā· low 13.
You now have a repeatable pipeline for real test data ā 40 validated, deduped, labeled rows\nwhere you used to have two.
Promptfoo reads test cases from a CSV: each column becomes a variable, and the special\n__expected column holds a per-row assertion. We map ticket ā the prompt var and\n__expected ā icontains:<the correct category>. jq @csv handles the commas and quotes\ninside ticket text:
{ echo 'ticket,__expected'; jq -r '[.ticket, (\"icontains:\" + .category)] | @csv' tickets_dataset.jsonl; } > tests.csv
wc -l tests.csv # 41 = header + 40 rows
The dataset is the deliverable ā everything from here is what it unlocks. First payoff: a model\ncomparison you can actually believe. On two hand-typed cases a ranking is noise; on 40 real,\nlabeled rows it's a measurement. Add each WEC model as a provider and run the whole set:
\ncat > promptfooconfig.yaml <<'EOF'
description: \"Ticket classifier ā model matrix on the synthetic 40-row dataset\"
providers:
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen3.5-122B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
options:
transform: |
const m = output.match(/\\{[\\s\\S]*?\\}/);
return m ? m[0] : output;
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests: file://tests.csv
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
The result is the opposite of what \"bigger is better\" would predict:
\n| Model | Accuracy | Completion tokens / ticket |
|---|---|---|
Qwen2.5-3B-Instruct (non-reasoning) | 85% (34/40) | ~30 |
Qwen3.5:9B (reasoning) | 47.5% (19/40) | ~965 |
Qwen3.5-122B (reasoning) | 15% (6/40) | ~1,013 |
![\"Step](\"https://wec.wiline.com/docs/assets/images/dataset-eval-matrix-9ffe20345f2040803fce117f6803a6ff.png\")
\nFigure 4. Same dataset, three models ā the small non-reasoning model wins by a mile.
The small, non-reasoning model wins decisively ā and the bigger the reasoning model, the worse\nit does. The reasoning models aren't dumber at classifying; they bury the JSON in their\nchain-of-thought. Qwen3.5:9B leaks Thinking: ā¦ into the output; Qwen3.5-122B often returns\nan empty answer entirely ā all ~1,000 tokens spent reasoning, nothing parseable left. They also\nburn ~30Ć more tokens per ticket, which is why the full matrix took 12m 31s, almost all of\nit spent waiting on the two big models. For structured output, thinking out loud is a liability,\nnot an advantage ā a counterintuitive finding you'd never get from a leaderboard.
But notice what actually made that finding possible: the dataset. On two hand-typed cases those\npercentages would be coin-flips; 40 labeled rows are what turn \"which model?\" from a guess into a\nmeasurement. The comparison is the reward for building the data ā not a substitute for it.
\nYou can now pick a model with evidence instead of vibes ā something no leaderboard can do for\nyou, because it doesn't know your task.
(The transform grabs the first {ā¦} block with a non-greedy *? match ā cheap insurance for\nmodels that wrap JSON in prose, exactly like part 2.)
category here ā priority is your next assertionThe __expected column grades only the predicted category against the label. The dataset also\ncarries a priority label we don't score yet. To grade both fields at once, add a javascript\nassertion that parses the JSON output and compares priority too ā the exact same pattern, one more\ncheck. Scoring more of what you generated is the cheapest way to make an eval stricter.
Take the winner ā Qwen2.5-3B at 85% (34/40) ā and pull its 6 failing tickets, comparing each\nto its label. (__expected isn't stored as a variable ā Promptfoo consumes it as the assertion ā\nso read the ground truth from the source dataset.) Because the data is freshly generated, your exact\nfailures will differ, but they consistently split into two recognizable kinds ā and neither is\njust \"the model is dumb\". Representative examples:
![\"The](\"https://wec.wiline.com/docs/assets/images/dataset-report-541b1a9921d846d5338c1b1547c49bd5.png\")
\nFigure 5. The web report makes it easy to scan which rows failed and open each one.
| Ticket | Label | Why it's tricky |
|---|---|---|
| \"educational discounts for startupsā¦\" | other | Debatable ā it's about pricing, so billing is defensible |
| \"grant a junior dev IAM access to S3ā¦\" | account | Debatable ā access control vs infrastructure |
| \"feedback on the API documentationā¦\" | other | Debatable ā feedback vs technical |
| \"update the card and add a userā¦\" | account | Multi-intent ā genuinely two categories at once |
The rest are ordinary misclassifications. The real, day-to-day lessons:
\nCommit tickets_dataset.jsonl and tests.csv alongside your eval config, and regenerate/grow\nthe dataset as your product changes.
{ \"error\": { \"message\": \"Authentication Error, invalid API key\", \"code\": \"401\" } }
The key is expired, revoked, or malformed. Generate a fresh one in Inference ā API Keys,\nre-export it, and re-sanitize (tr -cd '[:print:]') in case the paste carried an invisible character.
This comes from [\"billing\", ...] | index(.category) ā inside the pipe, . is the array, so\n.category indexes the array. Use the IN() idiom instead:
jq -c 'select((.category|IN(\"billing\",\"technical\",\"account\",\"other\")|not))' tickets.jsonl
![\"The](\"https://wec.wiline.com/docs/assets/images/dataset-ts-jq-index-ec85a2581780363d791db73ff2f03fd0.png\")
\nFigure 6. The index(.field) mistake ā inside the pipe, jq tries to index the array with a string.
nullāA null row means a generation call failed but the loop still wrote its (empty) result. Usual\ncauses: an expired/revoked API key, or the model timing out. Fixes: extract with\n.choices[0].message.content // empty so a failed call writes nothing instead of null;\nre-check wc -l after every generation run; and if calls hang, add a per-request timeout\n(curl -m 60) and consider a lighter, faster model for bulk generation.
__expected is null in the results JSONāThat's expected ā Promptfoo consumes the __expected CSV column as the row's assertion, so\nit isn't kept as a variable. To inspect ground-truth labels for failing rows, read them from your\nsource dataset (tickets_dataset.jsonl) rather than the results file.
You now have a real dataset ā but it runs in CI, offline. The next step is watching your app in\nproduction: self-hosting Langfuse to trace every call, track latency and cost, and run\nevals (using this dataset) against live traffic. That's where offline testing becomes real\nobservability ā and it's the next post in the series.
", "url": "https://wec.wiline.com/docs/tutorials/synthetic-eval-datasets-wiline-inference/", "title": "Stop hand-writing test cases: generate an eval dataset with the WEC API", "summary": "Two hand-typed test cases aren't an eval. Use the WEC Inference API to generate a labeled evaluation dataset ā then validate and curate it, because generated labels aren't automatically correct. The result is real test data at scale; feed it to your harness and coverage surfaces the misclassifications and debatable labels two cases would hide. Every command and result is real.", "date_modified": "2026-07-06T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "evals", "promptfoo", "inference", "datasets", "synthetic-data", "observability" ] }, { "id": "https://wec.wiline.com/docs/tutorials/structured-output-evals-wiline-inference/", "content_html": "++\"Return only JSON\" is one of the most common instructions in production LLM apps ā and one\nof the least reliable. Models wrap JSON in markdown fences, add a friendly sentence, or (if\nthey're reasoning models) narrate their entire thought process around it. Any of those\nbreaks a strict JSON.parse, and your pipeline falls over.
In this guide we build a Promptfoo eval that makes WEC\nInference API models classify support tickets into schema-validated JSON, then harden\nit against real-world messiness ā and use it to pick a model you can actually trust. This is\npart 2 of the evals series (see part 1\nfor first-run setup). Everything here was run live against https://inference.wiline.com.
Every test runs through the same pipeline. The detail that matters most: the transform\nruns on the model's output before the assertion sees it ā that's what lets us recover\nmessy JSON before validating it.
A WEC Inference API key ā see Inference ā API Keys.
\nThe model names used here (GLM-5.2, gemma4, Qwen2.5-3B-Instruct, qwen3.5:9B) come\nfrom the Models Hub ā swap in whatever you've enabled.
Node.js 22+ and your key exported (Promptfoo runs via npx):
nvm use 24
export WEC_API_KEY='sk-your-key'
If your first run errors with Cannot convert argument to a ByteString, your pasted key has\nan invisible character ā see Troubleshooting at the bottom.
We'll do something closer to a real workload than \"capital of France\": classify a support\nticket into category, priority, and a short summary. The key upgrade over a basic\ncheck is the JSON Schema ā it enforces enums, so a model that invents a category fails.
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā structured output (JSON schema)\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
The result ā 50%, and the failure is instructive:
\nticket [zai-org/GLM-5.2]
I was charged twice for my instanceā¦ [FAIL] ```json
{ \"category\": \"billing\", \"priority\": \"high\", ā¦ }
```
My VM won't boot after the latestā¦ [PASS] { \"category\": \"technical\", ā¦ }
ā 1 passed (50%) ā 1 failed (50%)
![\"Step](\"https://wec.wiline.com/docs/assets/images/json-step1-schema-fail-dc2b9c12c0ee29b0086499d5c76f2fd6.png\")
\nFigure 1. Same model, same prompt, temperature 0 ā one response is clean, the other is\nwrapped in ```json fences, so is-json can't parse it.
The content is correct; the format isn't. is-json validates both \"is it parseable JSON\"\nand \"does it match the schema\" ā and fenced output fails the first test.
transformāA transform runs on the model's output before assertions. Here we strip the fences.\n(This block uses four backticks because the config itself contains triple backticks.)
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā fence-stripping transform\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
options:
transform: |
return output.replace(/```json\\n?|\\n?```/g, '').trim();
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
Now both pass ā 2/2. The fences are stripped before is-json sees the output.
![\"Step](\"https://wec.wiline.com/docs/assets/images/json-step2-transform-pass-14cb388c6725d0aacffeee1bb320f553.png\")
\nFigure 2. The transform removes the code fences, so the previously-failing ticket parses.
transform needs an explicit returnA multi-line transform is a function body, not an expression ā omit return and you get\nTransform function did not return a value. See Troubleshooting.
Rather than clean up after the model, ask the API to only emit valid JSON via\nresponse_format. No transform this time:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā JSON mode (response_format)\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
response_format: { type: json_object }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
2/2 PASS with no transform ā looks like JSON mode works.
\n![\"Step](\"https://wec.wiline.com/docs/assets/images/json-step3-jsonmode-6cabcc01d214aaa3b14b1a0842eeb720.png\")
\nFigure 3. With response_format: json_object, GLM-5.2 returns clean JSON, no transform needed.
But before trusting response_format, we need to test it across models ā because a single\ngreen result can be luck, not enforcement.
Run the JSON-mode config against all four chat models. This is the payoff ā a\nWEC-specific reliability comparison you can't get from a generic leaderboard.
\ncat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā JSON-mode reliability across models\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0, response_format: { type: json_object } }
- id: openai:chat:gemma4
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0, response_format: { type: json_object } }
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0, response_format: { type: json_object } }
- id: openai:chat:qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0, response_format: { type: json_object } }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
4/8. GLM-5.2 and Qwen2.5-3B pass cleanly; gemma4 and qwen3.5:9B dump their reasoning\n(\"Thinking Processā¦\") instead of JSON ā despite response_format being set.
![\"Step](\"https://wec.wiline.com/docs/assets/images/json-matrix-responseformat-d227b3a1ca8bffbf85f89ca2da11eb87.png\")
\nFigure 4. response_format set on all four ā but the two reasoning-oriented models ignore it.
| Model | Result | Tokens (2 calls) |
|---|---|---|
zai-org/GLM-5.2 | ā ā | 202 |
Qwen2.5-3B-Instruct | ā ā | 243 |
gemma4 | āā (reasoning) | 251 |
qwen3.5:9B | āā (reasoning) | 1,757 |
You can now measure which models actually honor JSON mode ā instead of trusting the flag.
response_format doing anything?āDon't assume the reasoning was caused by JSON mode. Re-run the same matrix without\nresponse_format (using the fence-strip transform instead) and compare:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā matrix without response_format\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:gemma4
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
options:
transform: |
return output.replace(/```json\\n?|\\n?```/g, '').trim();
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
Still 4/8 ā the exact same models fail. So response_format changed nothing: on WEC\ntoday it's effectively a no-op (the backend isn't enforcing JSON mode). The reasoning\nbehavior belongs to the models, not to the flag.
![\"Step](\"https://wec.wiline.com/docs/assets/images/json-matrix-transform-77179c366ad47c9eb5b00fe2b5d92381.png\")
\nFigure 5. Identical result without JSON mode ā the flag wasn't doing anything.
Lesson worth its own callout: don't attribute a result to a change until you've tested the\nchange in isolation. One extra run turned a wrong conclusion into the right one.
The reasoning models do emit JSON ā it's just buried in prose. Instead of stripping fences,\nextract the first { ā¦ } block from anywhere in the output:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API ā JSON extraction transform\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:gemma4
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
- |
Classify this support ticket. Return ONLY JSON with keys:
\"category\" (one of: billing, technical, account, other),
\"priority\" (one of: low, medium, high),
\"summary\" (string, max 12 words).
Ticket: {{ticket}}
defaultTest:
options:
transform: |
const m = output.match(/\\{[\\s\\S]*\\}/);
return m ? m[0] : output;
assert:
- type: is-json
value:
type: object
required: [category, priority, summary]
properties:
category: { type: string, enum: [billing, technical, account, other] }
priority: { type: string, enum: [low, medium, high] }
summary: { type: string }
tests:
- vars: { ticket: \"I was charged twice for my instance this month.\" }
assert: [{ type: icontains, value: billing }]
- vars: { ticket: \"My VM won't boot after the latest snapshot restore.\" }
assert: [{ type: icontains, value: technical }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
7/8 ā a big jump. gemma4 and Qwen2.5 fully recover; qwen3.5:9B recovers one of two. The\none remaining failure is telling: that response emitted JSON and then kept reasoning (with\nmore braces), so the greedy { ā¦ } match over-captured and produced invalid JSON.
![\"Step](\"https://wec.wiline.com/docs/assets/images/json-matrix-extraction-3b34ee4a32f6990e447d9590d51d8911.png\")
\nFigure 6. Extracting the JSON block recovers the chatty models ā but it's a heuristic, not\na guarantee (note the one over-capture failure).
The terminal table is fine for a quick read, but the web report is where a matrix like this\ncomes alive:
\nnpx -y promptfoo@latest view
The dashboard at the top gives you the model-reliability picture at a glance ā the Pass\nRate (%) bars are the ones to watch here: three models tall, qwen3.5:9B sitting at 50%.\nBelow, each provider is a column you can expand to the full response (that's how we pulled\nthe over-capture failure in Figure 10 ā the terminal truncates it).
![\"The](\"https://wec.wiline.com/docs/assets/images/json-view-report-3739e5abd8419ffc8ece8271ff559851.png\")
\nFigure 7. The report's dashboard ā the Pass Rate bars turn the model matrix into a\none-glance ranking, and any cell expands to the full output.
The report's other charts (the prompt-vs-prompt scatter) shine when you're A/B-testing\nprompts, not models ā a good subject for a later post. For a single-prompt, multi-model run\nlike this one, the Pass Rate bars are the chart that matters.
Promptfoo exits non-zero when tests fail, so a one-liner is a gate (put it in a script or\nMakefile, never your interactive shell):
npx -y promptfoo@latest eval -c promptfooconfig.yaml \\
&& echo \"ā
passed\" \\
|| { echo \"ā eval failed ā blocking\"; exit 1; }
![\"The](\"https://wec.wiline.com/docs/assets/images/json-ci-gate-71912b506157c96cfd7daa8e5c81f300.png\")
\nFigure 8. A real failure blocks: 7/8 passed, but qwen3.5:9B's over-capture (from Step 6)\ntrips the gate ā the non-zero exit code is the signal CI uses to block a PR.
On GitLab:
\neval:
image: node:22
script:
- npx -y promptfoo@latest eval -c promptfooconfig.yaml
# WEC_API_KEY set as a masked CI/CD variable ā never committed.
Schema-validated JSON is now a CI gate ā a model or prompt change that breaks parsing\nnever ships.
GLM-5.2 and Qwen2.5-3B reliably produce schema-valid JSON\nand are ~8Ć cheaper than the reasoning models. For structured output on WEC, start there.qwen3.5:9B, and gemma4 on the day we tested) narrate their thinking\nand cost far more ā poor fits for strict JSON.transform is your safety net ā fence-stripping recovers the common case; JSON\nextraction recovers most chatty output ā but extraction is a heuristic that can over-capture.response_format is a no-op on WEC today ā verified by isolating it. Don't rely on it;\nvalidate instead.A multi-line transform is compiled as a function body, so it needs an explicit\nreturn. output.replace(...) on its own returns nothing.
options:
transform: |
return output.replace(/.../g, '').trim(); # note the `return`
![\"The](\"https://wec.wiline.com/docs/assets/images/json-ts-transform-return-764896c26bc05d65d37eac7808ee06fb.png\")
\nFigure 9. Without return, Promptfoo errors instead of transforming.
If is-json fails with \"Expected output to be valid JSON\" and the response contains a\n\"Thinking Processā¦\" / \"Final Reviewā¦\" narration, you're using a reasoning model. Sometimes it\nemits only reasoning; sometimes ā as below ā it emits valid JSON and then keeps talking,\nso even the Step 6 extraction over-captures and the parse fails. Options: switch to\nGLM-5.2 / Qwen2.5-3B, or accept that the extraction transform is best-effort, not a guarantee.
![\"The](\"https://wec.wiline.com/docs/assets/images/json-ts-reasoning-dump-dbfebf9e8d4f5f89e6b2e14b8e0edf50.png\")
\nFigure 10. The failing case expanded in the report: qwen3.5:9B returned valid JSON and\nthen kept reasoning (\"7. Final Reviewā¦\"), so the extraction over-captured and the assertion\nreports \"Expected output to be valid JSON.\"
Your pasted API key contains an invisible Unicode character. Strip it:
\nexport WEC_API_KEY=$(printf '%s' \"$WEC_API_KEY\" | LC_ALL=C tr -cd '[:print:]')
![\"The](\"https://wec.wiline.com/docs/assets/images/json-ts-bytestring-0bb4979ce94f0775f1fe314315b38e01.png\")
\nFigure 11. A stray character in the key trips the HTTP header before any request is sent.
Have a structured-output case that still slips through? That's the next test to add.
", "url": "https://wec.wiline.com/docs/tutorials/structured-output-evals-wiline-inference/", "title": "Trustworthy JSON: schema-validate your model's structured output", "summary": "LLMs promise JSON and deliver markdown fences and reasoning. Build a Promptfoo eval that classifies support tickets into schema-validated JSON on the WEC Inference API ā with transforms to recover messy output, a model-reliability matrix, and a CI gate. Every command and result is real.", "date_modified": "2026-07-01T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "evals", "promptfoo", "inference", "json", "structured-output", "observability" ] }, { "id": "https://wec.wiline.com/docs/tutorials/eval-models-promptfoo-wiline-inference/", "content_html": "\n+You wouldn't ship code without tests ā but most teams ship LLM features on \"looks good\nto me.\" Evaluations (evals) fix that: you define test cases and pass/fail criteria, then\nmeasure your model objectively ā catching regressions, comparing models, and gating\ndeploys.
\nIn this guide you'll build a real eval harness with Promptfoo\npointed at the WEC Inference API ā the same OpenAI-compatible endpoint you call from\nyour apps. Everything here was run live against https://inference.wiline.com; the outputs\nare real.
A WEC (WiLine Edge Cloud) account and an Inference API key ā see\nInference ā API Keys.
\nNode.js 22+. Promptfoo treats Node 20 as end-of-life; with nvm:
nvm install 24 # 24 LTS preferred
nvm use 24
node -v # v24.x
We'll run Promptfoo with npx ā nothing to install globally.
![\"The](\"https://wec.wiline.com/docs/assets/images/promptfoo-api-key-a1aca6bfffa7335651ab59ec1c07d8f4.png\")
\nFigure 1. Create a key under Inference ā API Keys, then copy it.
Put your key in the shell:
\nexport WEC_API_KEY='sk-your-key-here'
If your first eval errors with Cannot convert argument to a ByteString, see Troubleshooting #1 below.
The API is OpenAI-compatible, so the model list is a plain GET /v1/models:
curl -s https://inference.wiline.com/v1/models \\
-H \"Authorization: Bearer $WEC_API_KEY\" | jq -r '.data[].id'
zai-org/GLM-5.2
gemma4
Qwen2.5-3B-Instruct
qwen3.5:9B
whisper-large-v3
whisper-medium
whisper-base
The whisper-* models are speech-to-text; the rest are chat models. We'll evaluate the\nchat models, starting with zai-org/GLM-5.2.
Create a config. apiBaseUrl points Promptfoo at WEC; apiKeyEnvar tells it which env\nvar holds your key:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- \"Answer in a single word. What is the capital of {{country}}?\"
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-first-eval-42c0a9a515e6a1e5af7a6cf5c6a30118.png\")
\nFigure 4. Your first passing eval against the WEC Inference API.
icontains is a deterministic assertion ā a case-insensitive substring check. Cheap,\nfast, and perfect for facts with a known answer.
npx -y promptfoo@latest view
This opens a local web UI with every prompt, output, and pass/fail ā far easier to scan\nthan the terminal once you have more than a handful of cases.
\n![\"The](\"https://wec.wiline.com/docs/assets/images/promptfoo-view-report-b93aa9bcadc4c60e008df1bb5effcd3c.png\")
\nFigure 5. promptfoo view ā the same results in the browser.
Adding a provider is one extra block. Here we add Qwen2.5-3B-Instruct alongside GLM-5.2\nand run the same two test cases against both:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā model comparison\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
- id: openai:chat:Qwen2.5-3B-Instruct
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- \"Answer in a single word. What is the capital of {{country}}?\"
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-two-model-3c6c51ea8839c03dd51d270c78648ec9.png\")
\nFigure 6. Both models pass. GLM-5.2's results are served from cache (0 new requests); only Qwen2.5 makes live calls.
Substring checks can't judge an open-ended answer. A model-graded llm-rubric assertion\nuses a model to score the output against criteria ā and the grader can run on WEC too, so\nyou need no external provider:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā model-graded (LLM rubric)\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
defaultTest:
options:
provider: # the grader model
id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
prompts:
- \"Explain what a {{topic}} is in two sentences, for a beginner.\"
tests:
- vars: { topic: VPC }
assert:
- type: llm-rubric
value: \"Describes a VPC as a private, isolated virtual network in the cloud, and is understandable to a beginner.\"
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-llm-rubric-f969ad98a43427eacf9f83f13f2e6dde.png\")
\nFigure 7. The run shows two token lines ā Eval (75) and Grading (350) ā because the rubric check is itself a model call.
Model-graded checks are powerful, but they aren't free.
\nA correct answer that takes 30 seconds is still a production problem. Add a latency\nguardrail to defaultTest (it applies to every case). Always run with --no-cache ā cached\nresponses report ~0 ms and would pass the gate for free:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā latency guardrail\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
defaultTest:
assert:
- type: latency
threshold: 5000 # fail anything slower than 5 s
prompts:
- \"Answer in a single word. What is the capital of {{country}}?\"
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
This failed ā the answers were correct (Paris, Tokyo) but both calls took ~33 s,\ntripping the 5 s gate:
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-latency-fail-ab40e8dc4386c6acd496942ee3a490aa.png\")
\nFigure 8. The guardrail fires: correct answers, wrong latency. Duration: 33s.
Bump the threshold to something realistic and re-run:
\nsed -i 's/threshold: 5000/threshold: 60000/' promptfooconfig.yaml
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-latency-pass-f3fa1a1c9cb0a11a87328b2a15ef1f31.png\")
\nFigure 9. With a 60 s threshold the same calls pass easily ā Duration: 2s.
Back-to-back calls to the same model ranged from ~2 s to ~33 s on the same day. Run a\nfew --no-cache passes to get a realistic baseline before locking in your threshold.
The cost assertion doesn't work against WEC (see Troubleshooting #2). Use latency + max_tokens together as your production guardrail instead:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā guardrails (latency + token cap)\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
max_tokens: 16
defaultTest:
assert:
- type: latency
threshold: 30000
prompts:
- \"Answer in a single word. What is the capital of {{country}}?\"
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml --no-cache
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-latency-token-cap-968bf5b53dd3edf6d77129f011e4a66f.png\")
\nFigure 11. With a 30 s threshold and a 16-token cap, both cases pass.
Real apps want JSON, not prose. is-json checks the output is valid JSON and matches a\nschema in one assertion:
cat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā JSON schema validation\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config:
apiBaseUrl: https://inference.wiline.com/v1
apiKeyEnvar: WEC_API_KEY
temperature: 0
prompts:
- 'Return ONLY a JSON object (no markdown, no prose) with keys \"capital\" (string) and \"population_millions\" (number) for the country {{country}}.'
defaultTest:
assert:
- type: is-json
value:
type: object
required: [capital, population_millions]
properties:
capital: { type: string }
population_millions: { type: number }
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-json-schema-82dcc59881029f9bc0134dcf5a5c5ea2.png\")
\nFigure 12. GLM-5.2 returns compact, schema-valid JSON on the first try.
Here's what you can't get anywhere else ā the same strict-JSON test across all the chat\nmodels on your account:
\ncat > promptfooconfig.yaml <<'EOF'
description: \"WEC Inference API eval ā model matrix\"
providers:
- id: openai:chat:zai-org/GLM-5.2
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:gemma4
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:Qwen2.5-3B-Instruct
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
- id: openai:chat:qwen3.5:9B
config: { apiBaseUrl: https://inference.wiline.com/v1, apiKeyEnvar: WEC_API_KEY, temperature: 0 }
prompts:
- 'Return ONLY a JSON object (no markdown, no prose) with keys \"capital\" (string) and \"population_millions\" (number) for the country {{country}}.'
defaultTest:
assert:
- type: is-json
value:
type: object
required: [capital, population_millions]
properties:
capital: { type: string }
population_millions: { type: number }
tests:
- vars: { country: France }
assert: [{ type: icontains, value: Paris }]
- vars: { country: Japan }
assert: [{ type: icontains, value: Tokyo }]
- vars: { country: Brazil }
assert: [{ type: icontains, value: Bras }]
EOF
npx -y promptfoo@latest eval -c promptfooconfig.yaml
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-model-matrix-table-fcebe135ddd276ef881fc83a726c6b96.png\")
\nFigure 13a. 12 test cases across 4 models ā 9 pass, 3 fail. qwen3.5:9B emits its chain-of-thought instead of JSON.
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-model-matrix-summary-96e3ce6b439ae8d487aef486120dbe97.png\")
\nFigure 13b. The token breakdown tells the real story: the reasoning model cost 13Ć more for the same task.
| Model | Strict JSON? | Notes | Tokens (3 calls) |
|---|---|---|---|
zai-org/GLM-5.2 | ā all pass | compact JSON | 177 |
gemma4 | ā all pass | integer populations | 207 |
Qwen2.5-3B-Instruct | ā all pass | pretty-printed, precise | 262 |
qwen3.5:9B | ā all fail | reasoning model ā emits its \"thinking\" instead of raw JSON | 2,650 |
Two lessons fall out immediately:
\nqwen3.5:9B is a reasoning model. It \"thinks out loud,\" so it breaks strict JSON\nunless you strip the reasoning or use a response-format constraint. Picking it for\nstructured output would silently fail.That's a model-selection decision you can now defend with data ā for your workloads, not a\ngeneric leaderboard.
\nYou can now compare every model on your endpoint with objective, repeatable numbers ā\naccuracy, latency, and cost side by side.
Promptfoo exits non-zero (100) when any test fails. Verify this first:
npx -y promptfoo@latest eval -c promptfooconfig.yaml > /dev/null 2>&1; echo \"exit code: $?\"
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-exit-code-a3471bde65e7ebf24feaf0d964a3b7eb.png\")
\nFigure 14. Exit code 100 confirms Promptfoo signals failure reliably ā exactly what CI needs.
Wire it up as a gate:
\nnpx -y promptfoo@latest eval -c promptfooconfig.yaml \\
&& echo \"ā
eval passed\" \\
|| { echo \"ā eval failed ā blocking\"; exit 1; }
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-ci-gate-table-7067b62a9b737ed79e3906b6da54ea2a.png\")
\nFigure 15a. The gate re-runs the matrix eval from cache ā same 9/3 result.
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-ci-gate-result-8765593b4f037c480e7fcef844ff6eb4.png\")
\nFigure 15b. Promptfoo exits non-zero, the || branch fires, and the gate blocks.
Never run the exit 1 variant in your interactive shell ā see Troubleshooting #3.
In CI, you don't even need the exit 1 ā the non-zero exit fails the job. On GitLab:
eval:
image: node:22
script:
- npx -y promptfoo@latest eval -c promptfooconfig.yaml
# WEC_API_KEY is set as a masked CI/CD variable ā never commit it.
Now a prompt change that quietly breaks JSON output, or a model swap that doubles latency,\nfails the pipeline before it reaches users.
\nYour eval is now a deploy gate ā regressions fail CI instead of reaching users.
API call error: TypeError: Cannot convert argument to a ByteString because
the character at index 32 has a value of 8232 which is greater than 255.
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-invisible-char-error-540ad4b2ab9242c120ae80870319df01.png\")
\nFigure 2. Every API call errors before reaching the server ā the key never even leaves the machine.
Cause: The pasted key contains an invisible Unicode character ā U+2028 (line separator). Some terminals and clipboard managers insert it silently when copying from certain sources.
\nFix: Strip non-printable characters from the key:
\nexport WEC_API_KEY=$(printf '%s' \"$WEC_API_KEY\" | LC_ALL=C tr -cd '[:print:]')
printf 'len=%s\\n' \"$(printf %s \"$WEC_API_KEY\" | wc -c)\"
# len=25
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-key-strip-1a9c78d8c230c69db3e89eecfdfa24ca.png\")
\nFigure 3. After stripping, the key length confirms no extra characters remain. Re-run the eval ā it passes.
Error: Cost assertion does not support providers that do not return cost
![\"Terminal:](\"https://wec.wiline.com/docs/assets/images/promptfoo-cost-error-1615b4a5c8b0973274d32c6d658a2e12.png\")
\nFigure 10. The WEC Inference API doesn't return a cost field in its responses.
Cause: Promptfoo calculates cost by looking up a price-per-token for the model in its built-in price list and multiplying by the tokens used. It has prices for the big hosted models but not for WEC's self-hosted model names, so it has no price to apply and the assertion fails. (Most APIs, including OpenAI, don't return a dollar cost in the response ā tools compute it.)
\nFix: Skip the cost assertion entirely. Track spend from the Total Tokens line Promptfoo prints at the end of every run, and use max_tokens in the provider config to cap per-call token usage.
exit 1 closes your SSH sessionāā eval failed ā blocking
logout
Connection to 10.80.4.212 closed.
Cause: Running || { echo \"ā eval failed ā blocking\"; exit 1; } directly in an interactive shell exits the shell process itself ā which, over SSH, closes the connection.
Fix: Put the gate in a non-interactive context: a Makefile target, an npm script, a git pre-commit hook, or a CI job. In those environments exit 1 fails the job without touching your terminal. If you need to test the gate locally, use || echo \"ā eval failed\" (without exit 1) to see the output safely.
You've gone from \"looks good to me\" to a real, automatable eval harness on WEC ā with\ndeterministic checks, guardrails, schema validation, model-graded rubrics, a model matrix,\nand a CI gate.
\nComing up in this series:
\nllm-rubric, factuality scoring, and\nmulti-judge checks so your grader is trustworthy.+Through Parts 1ā4 you deployed\nOpenClaw, secured it with HTTPS, added Telegram, and made it private over a NetBird\nmesh ā all pointed at OpenAI. This part swaps the model out from under it:\npoint the same agent at WEC Models ā WiLine's own OpenAI-compatible inference ā\nand run it on an open-weight model, Llama 3.1 8B Instruct. Same box, no rebuild\nā just a base-URL, key, and model change via the OpenClaw CLI.
\n\nContinues on the same WEC Instance from Parts 1ā4. WEC Models is an\nOpenAI-compatible endpoint at https://inference.wiline.com/v1; every command\nbelow was run against a live OpenClaw gateway container\n(openclaw-openclaw-gateway-1).
If you self-hosted the agent to begin with, you already care about control ā not\ndepending on someone else's cloud, not being one pricing-page email away from a\nrebuild. But there's a gap in that story: the agent runs on your box, and every\nsingle message it handles is still shipped off to OpenAI to think. Self-hosting\nthe agent and outsourcing the model is only half the job. This part closes it.
\nIt's also just cheaper. WEC's own catalog prices Llama3.1-8B-Instruct at\n$0.15 / 1M input tokens Ā· $0.20 / 1M output tokens. OpenAI doesn't publish a\nstandalone rate for gpt-5.5 (the model OpenClaw defaults to in\nPart 1) any more, but its closest\ncurrent family member ā GPT-5.6 \"Terra,\" pitched by OpenAI as\n\"performance competitive with GPT-5.5\" ā lists at\n$2.50 / 1M in Ā· $15 / 1M out. That's an approximate comparison, not an\napples-to-apples benchmark ā the honest move, as always on this blog, is to\neval it on your own workload\nrather than trust either vendor's number. But the direction is clear enough to be\nworth the ten minutes this takes.
openclaw-openclaw-gateway-1.See Inference ā API Keys if you\ndon't already have one (the same key from\nPart 1 of the evals series\nworks fine here).
\n\nFigure 1. Create a key under Inference ā API Keys, then copy it.
Keep it out of your shell history ā put it in a file instead of exporting it\ndirectly:
\necho 'WEC_API_KEY=sk-wec-...' >> env.local
source <(grep -v '^#' env.local | sed 's/^/export /')
Query the OpenAI-compatible /v1/models endpoint to see what's live on your WEC\nInstance:
curl -s -H \"Authorization: Bearer $WEC_API_KEY\" https://inference.wiline.com/v1/models
The response lists every model available to your key, alongside the id this\ntutorial uses:
\n{\"data\":[
...,
{\"id\":\"Llama3.1-8B-Instruct\",\"object\":\"model\",\"created\":1677610602,\"owned_by\":\"openai\"},
...
]}
Model ids on WEC aren't always the display name shown in the portal ā always take\nthe exact id from this response, not a guess.
Don't hand-edit openclaw.json ā use openclaw config set. It validates the\nwhole file on every write, so a typo fails loudly instead of quietly breaking\nthe gateway. A custom provider needs its base URL, its API style, and its model\nlist declared together, in one write ā the schema rejects a provider that's\nmissing any of the three, so build the full object up front:
docker exec openclaw-openclaw-gateway-1 openclaw config set models.providers.wec '{
\"baseUrl\": \"https://inference.wiline.com/v1\",
\"api\": \"openai-completions\",
\"models\": [
{\"id\": \"Llama3.1-8B-Instruct\", \"name\": \"Llama 3.1 8B Instruct\"}
]
}' --strict-json
Updated models.providers.wec. Change will apply without restarting the gateway.
That registers WEC as a provider, but it's not authenticated yet. Don't put the\nkey straight into that JSON ā OpenClaw's compose setup keeps API keys out of\nenvironment variables entirely and stores them in a dedicated, file-backed\nsecrets directory instead (OPENCLAW_AUTH_PROFILE_SECRET_DIR, mounted into the\ncontainer). Hand the key to OpenClaw's own auth store and let it do the wiring:
echo \"$WEC_API_KEY\" | docker exec -i openclaw-openclaw-gateway-1 openclaw models auth paste-api-key --provider wec
Updated config: $OPENCLAW_HOME/.openclaw/openclaw.json
Backup: $OPENCLAW_HOME/.openclaw/openclaw.json.bak
Auth profile: wec:manual (wec/api_key)
Check that the provider and its auth resolved together, then make the model\nthe default so every agent picks it up without extra config:
\ndocker exec openclaw-openclaw-gateway-1 openclaw models list
docker exec openclaw-openclaw-gateway-1 openclaw models set wec/Llama3.1-8B-Instruct
Model Input Ctx Local Auth Tags
wec/Llama3.1-8B-Instruct text 200k no yes configured
openai/gpt-5.5 text 200k no yes configured,alias:GPT
...
Updated config: $OPENCLAW_HOME/.openclaw/openclaw.json
Backup: $OPENCLAW_HOME/.openclaw/openclaw.json.bak
Default model: wec/Llama3.1-8B-Instruct
![\"Terminal](\"https://wec.wiline.com/docs/assets/images/openclaw-wec-models-list-bb96e6c9a1dfe7d0c98be4a6b5b40ae6.png\")
\nFigure 2. wec/Llama3.1-8B-Instruct, authenticated and set as the default model.
OpenClaw can run more than one agent per gateway, so every message needs a\ntarget ā pass --agent, matching whatever openclaw agents list calls yours\n(main, unless you renamed it):
docker exec openclaw-openclaw-gateway-1 openclaw agents list
Agents:
- main (default)
Workspace: $OPENCLAW_HOME/.openclaw/workspace
Model: wec/Llama3.1-8B-Instruct
Routing rules: 0
Send it a message and see who answers:
\ndocker exec openclaw-openclaw-gateway-1 openclaw agent --agent main --message \"What is the capital of France?\"
Paris is the capital of France.
That's WEC, on Llama 3.1, with no OpenAI in the loop.
\n![\"Terminal](\"https://wec.wiline.com/docs/assets/images/openclaw-wec-agent-verified-e5125e1d417f258ba73f9d11b78244ff.png\")
\nFigure 3. The agent, running on Llama 3.1 8B Instruct via WEC.
Want it snappier? Turn off thinking mode ā it's a slash command inside the\nchat, not a shell command:
\ndocker exec openclaw-openclaw-gateway-1 openclaw agent --agent main --message \"/think off\"
Thinking mode has been turned off.
Or make it stick for every session:
\ndocker exec openclaw-openclaw-gateway-1 openclaw config set agents.defaults.thinkingDefault \"off\"
Updated agents.defaults.thinkingDefault. No gateway restart needed.
\"custom model providers must declare models\" ā declare baseUrl, api,\nand models together, in one config set call. A custom provider missing any\nof the three gets rejected; you can't add models in a follow-up write.
\"models.providers.wec.apiKey is unresolved in the active runtime snapshot\"\nā don't set apiKey on the provider object; let models auth paste-api-key\nown it (Step 3). Already set one? Remove it:
docker exec openclaw-openclaw-gateway-1 openclaw config unset models.providers.wec.apiKey
\"No target session selected\" ā always pass --agent <id> with\nopenclaw agent. Check the id with openclaw agents list first if you're not\nsure (main, unless you renamed it).
-bash: /think: No such file or directory ā /think is a slash command\nfor inside the chat, not a shell command. Pass it through --message instead,\nas in Step 5.
With the agent running on our own model, the next step is measuring it ā\nevaluating answer quality with Promptfoo + WEC Models.
\n+![\"SQLite\"](\"https://wec.wiline.com/docs/assets/images/SQLite370-e02c30e84f9e098787bb57701b0b7232.png\")
Hermes Agent is Nous Research's open-source\n(MIT) AI agent ā \"the agent that grows with you.\" Its standout feature is\npersistent memory: it learns your projects and doesn't forget across\nrestarts. This guide deploys it on the same WEC Instance you already use for\nOpenClaw, points it at a model, and proves the memory survives a full reboot.
\n\nRuns on the same WEC Instance from the OpenClaw series ā no new VM. The installer\npulls its own Python (via uv), Node.js browser tools, and a Playwright Chromium\nautomatically. State lives in ~/.hermes/. The messaging gateway runs as a systemd\nuser service, so it coexists with the OpenClaw Docker stack. Captured live on Ubuntu\n22.04 with hermes v0.17.0.
An agent that forgets on every restart is a toy. Hermes keeps a durable store on\ndisk under ~/.hermes/:
~/.hermes/memories/MEMORY.md ā a human-readable file of facts the agent has\nchosen to remember.~/.hermes/state.db ā a SQLite database holding every session and message,\nwith FTS5 full-text search indexes layered on top so the agent can search its own\nhistory.A deploy, crash, or reboot doesn't wipe what it learned. That's the difference between\na chat demo and an agent you can rely on day to day ā and it's exactly what we test at\nthe end.
\nEverything runs on one WEC Instance. Hermes talks to a model over the WEC Inference API,\nkeeps its memory on local disk, and runs as a systemd user service ā so it restarts itself\nafter a reboot and its memory is still there:
\n\nglm-5.2 works too).curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
![\"Running](\"https://wec.wiline.com/docs/assets/images/hermes-install-start-1f3d29fcdd9d66dd1e5c1bc28b3c9e5d.png\")
The installer resolves a Python virtual environment with uv, installs the Node.js\nbrowser tools, downloads a Playwright Chromium, syncs the bundled skills, and drops a\nhermes launcher into ~/.local/bin:
ā Main package installed (hash-verified via uv.lock)
ā All dependencies installed
ā Browser engine setup complete
ā Installed hermes launcher ā ~/.local/bin/hermes
ā Configuration directory ready: ~/.hermes/
ā Starting setup wizard...
It then launches the setup wizard automatically. Reload your shell afterward so the\nhermes command is on your PATH:
source ~/.bashrc
The wizard walks through setup type, terminal backend, and messaging. On a box that\nalready ran OpenClaw, it offers to import the old config first ā you can preview\nwhat would be migrated before anything changes.
\nThe first choice is how to set up Hermes:
\n![\"The](\"https://wec.wiline.com/docs/assets/images/hermes-setup-choice-0db3a3f64faec3d72e3477b075af1901.png\")
We chose Quick Setup first. The Portal login showed a device code and sat at\n\"Waiting for approval (polling)\" ā it needs an active Portal subscription, which this\naccount didn't have. We pressed Ctrl+C to cancel and instead brought our own\nmodel in Step 3 (below) with an OpenAI key. If you do have a Portal plan, complete\nthe login here and you can skip Step 3.
For the rest of the wizard:
\nhermes setup gateway).The wizard applies sensible agent defaults (max iterations, a 50% context-compression\nthreshold) and finishes with a tool-availability summary.
\nOut of the box the default model routes through the Nous Portal / OpenRouter and needs\na key. To use your own provider instead, run:
\nhermes setup model
Then select OpenAI āø OpenAI API, paste your API key when prompted (it's written to\n~/.hermes/.env ā never echoed), accept the default base URL\nhttps://api.openai.com/v1, and pick a model ā we used gpt-5.5:
API key saved.
Base URL [https://api.openai.com/v1]:
Select default model:
ā (ā) gpt-5.5
![\"Choosing](\"https://wec.wiline.com/docs/assets/images/hermes-model-select-2b0c5452d2e217ff8692c022badd9b28.png\")
To run inference on WiLine's own endpoint, choose Custom endpoint, set the base URL\nto https://inference.wiline.com/v1, paste your WEC API key, and use a model like\nglm-5.2. Same OpenAI-compatible flow.
Confirm the model and key are healthy:
\nhermes doctor
ā Configuration Files
ā ~/.hermes/.env file exists
ā API key or custom endpoint configured
...
ā Memory Provider
ā Built-in memory active (no external provider configured ā this is fine)
The remaining ā lines in doctor are optional tool integrations (web search,\nDiscord, image gen) ā not needed for this guide.
Start an interactive session:
\nhermes
![\"The](\"https://wec.wiline.com/docs/assets/images/hermes-tui-welcome-842b22f5c733599e17b84cde75db4933.png\")
Send it a fact and ask it to remember ā watch for the š§ memory tool to fire:
ā Please save to your long-term memory: my WEC project codename is
Bluefin-7, running on the OpenClaw box.
ā š§ memory ? 0.0s
āā ā Hermes āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā®
Saved to long-term memory: your WEC project codename is Bluefin-7,
running on the OpenClaw box.
ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
![\"Hermes](\"https://wec.wiline.com/docs/assets/images/hermes-memory-save-458cefa8aaf46e0d1cc554418a3acc38.png\")
That write creates ~/.hermes/memories/MEMORY.md. Exit with /exit or Ctrl+D.
The TUI submits on newline, so pasting multi-line text can send a half-typed message.\nIf a prompt only catches a fragment, type it on one line by hand.
This is the whole point. The Hermes gateway is installed as a systemd user service\nwith linger enabled, so it comes back automatically after a reboot ā no login\nrequired. Restart the service, then reboot the whole box to make the test airtight:
\nsystemctl --user restart hermes-gateway # restart just the gateway service
sudo reboot # then reboot the entire box
![\"Restarting](\"https://wec.wiline.com/docs/assets/images/hermes-reboot-ae228254579427ae3018547803f83c62.png\")
Reconnect once the box is up and confirm the service restarted on its own:
\nsystemctl --user status hermes-gateway --no-pager
ā hermes-gateway.service - Hermes Agent Gateway - Messaging Platform Integration
Loaded: loaded (...; enabled; vendor preset: enabled)
Active: active (running) since Thu 2026-06-25 21:21:54 UTC; 1min 36s ago
![\"systemctl](\"https://wec.wiline.com/docs/assets/images/hermes-gateway-status-4da5472c10ec41fe81cfc64298589f10.png\")
Now open a fresh session and ask ā no prior context, brand-new session ID:
\nhermes
ā What's my WEC project codename, and which box does it run on?
āā ā Hermes āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā®
Your WEC project codename is Bluefin-7, and it runs on the
OpenClaw box.
ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
![\"A](\"https://wec.wiline.com/docs/assets/images/hermes-reboot-recall-eac2b093a48d12569fa730340f5751d3.png\")
It recalled the fact from disk after a full reboot. Inspect where that durable state\nlives:
\ncat ~/.hermes/memories/MEMORY.md
file ~/.hermes/state.db && du -h ~/.hermes/state.db
User's WEC project codename is Bluefin-7, running on the OpenClaw box.
/home/ubuntu/.hermes/state.db: SQLite 3.x database, ...
176K\t/home/ubuntu/.hermes/state.db
A plain-text MEMORY.md you can read and edit, backed by a SQLite state.db for\nsessions. That's the persistent memory ā and it just outlived a reboot.
When you ask Hermes to remember something, it writes the fact to\n~/.hermes/memories/MEMORY.md ā plain Markdown you can open, read, and even edit by\nhand. Separately, your full conversation history is kept in\n~/.hermes/state.db, a SQLite database. Both live on disk, which is why a\nreboot doesn't lose them.
The simplest way to see what the agent knows is just to read the Markdown:
cat ~/.hermes/memories/MEMORY.md
To look inside the database, note that SQLite's engine is built into Python, so Hermes\nuses it without any extra install. The optional sqlite3 command-line tool ā for\nbrowsing the DB by hand ā isn't installed by default. If you want it:
sudo apt install -y sqlite3
sqlite3 ~/.hermes/state.db '.tables'
![\"Output](\"https://wec.wiline.com/docs/assets/images/hermes-sqlite-tables-dbdc5016a3514137534038cac3f6298d.png\")
You'll see sessions and messages tables holding the conversation, plus a family of\nmessages_fts* tables. Those are SQLite FTS5 full-text indexes ā including a\ntrigram variant for fuzzy matching ā that let Hermes search its own past conversations:
CREATE VIRTUAL TABLE messages_fts USING fts5(content);
CREATE VIRTUAL TABLE messages_fts_trigram USING fts5(content, tokenize='trigram');
And each session is a row you can inspect ā here are the two from this guide, the\nsecond one created after the reboot:
sqlite3 -header -column ~/.hermes/state.db \\
\"SELECT substr(id,1,15) AS id, model, message_count FROM sessions ORDER BY started_at;\"
id model message_count
--------------- ------- -------------
20260625_211227 gpt-5.5 8
20260625_212333 gpt-5.5 2
You never need the CLI for Hermes to work ā it's only for inspecting the database\nyourself. Day to day, the human-readable MEMORY.md is all you have to look at.
You now run an agent whose memory survives reboots ā and you know exactly where that\nmemory lives on disk.
The messages table holds the actual text of every turn. Each row has a role ā\nuser (what you typed), assistant (Hermes's reply), or tool (a tool's result) ā so\nyou can replay the whole exchange:
sqlite3 -header -column ~/.hermes/state.db \\
\"SELECT substr(session_id,10,6) AS sess, role, substr(content,1,55) AS content
FROM messages ORDER BY timestamp;\"
sess role content
------ --------- -------------------------------------------------------
211227 user hello, can you hear me?
211227 assistant Yes ā I can hear you. How can I help?
211227 user Please save to your long-term memory: my WEC project...
211227 tool {\"success\": true, \"done\": true, \"target\": \"memory\", ...
211227 assistant Saved to long-term memory: your WEC project codename...
212333 user What's my WEC project codename, and which box does...
212333 assistant Your WEC project codename is Bluefin-7, and it runs...
![\"The](\"https://wec.wiline.com/docs/assets/images/hermes-sqlite-messages-12ab80eeb7db27fbf02d5c73b075d553.png\")
The interesting part is how the memory got written. The assistant didn't edit a file\ndirectly ā it emitted a memory tool call, and that tool is what wrote\nMEMORY.md. You can see the exact call recorded in the history:
sqlite3 ~/.hermes/state.db \\
\"SELECT tool_calls FROM messages WHERE tool_calls LIKE '%memory%' LIMIT 1;\"
{\"function\": {\"name\": \"memory\", \"arguments\":
\"{\\\"target\\\":\\\"memory\\\",\\\"operations\\\":[
{\\\"action\\\":\\\"add\\\",
\\\"content\\\":\\\"User's WEC project codename is Bluefin-7, running on the OpenClaw box.\\\"}]}\"}}
![\"The](\"https://wec.wiline.com/docs/assets/images/hermes-sqlite-memory-toolcall-53bc3762b14f9ca8e30715118341b792.png\")
So the full chain is: you type ā the agent decides it's worth keeping ā it calls the\nmemory tool with action: add ā that writes the fact into MEMORY.md, while the\nwhole conversation stays in state.db, full-text searchable via FTS5. Two stores, one\ndurable memory.
A memory that survives reboots only matters because of what the agent does with it.\nWith Hermes running, the same install ships these capabilities (you saw them in the\nhermes doctor tool list) ā here's what they unlock day to day:
terminal and code_execution tools let it act,\nnot just chat ā e.g. \"clone this repo, install deps, and run the tests,\" or \"check\nwhy the disk is filling up.\" It does it on the WEC Instance and reports back.cronjob tool runs recurring jobs: a morning\nbriefing, a weekly disk-usage report, or watch a service and alert you when\nit fails ā delivered to your messaging channel. (Part 3.)hermes setup tools.)The rest of this series turns each of these into a hands-on guide (see What's next).
\nQuick Setup (Nous Portal) hangs at \"Waiting for approval (polling)\". The Portal\nlogin needs an active subscription/credit. If you don't have one, Ctrl+C and use\nFull setup with your own key instead (Step 3).
doctor shows \"No API key found in ~/.hermes/.env\". The default model points at\nthe Portal/OpenRouter. Run hermes setup model, pick your provider, and paste the\nkey ā doctor then reports \"API key or custom endpoint configured.\"
Deprecated .env warning on gateway start (e.g. MESSAGING_CWD). Harmless;\nfollow the hint to move the setting into config.yaml when convenient.
Repeating WARNING [Telegram] ā¦ in the gateway logs. If you followed the earlier\nOpenClaw parts and added a Telegram channel, Hermes's setup detected OpenClaw and\nimported those messaging settings into ~/.hermes/.env. Because we skipped\nfinishing messaging here, the gateway has a token but no complete config, so it keeps\nretrying and logging this warning. (A fresh box with no OpenClaw Telegram setup won't\nsee it.) Remove the stray entries and restart:
sed -i '/^TELEGRAM_ALLOWED_USERS=/d; /^TELEGRAM_BOT_TOKEN=/d' ~/.hermes/.env
systemctl --user restart hermes-gateway
If that token was ever exposed, revoke it in Telegram's @BotFather and issue a new\none.
\nNow that Hermes is running with durable memory, the rest of the series turns each\ncapability above into a hands-on guide:
\nhermes setup gateway, allowlists, verify the bot\nresponds), so you can hand it tasks from your phone.Links will appear here as each part ships.
", "url": "https://wec.wiline.com/docs/tutorials/self-host-hermes-agent/", "title": "Self-host the Hermes Agent with persistent memory", "summary": "Deploy Nous Research's open-source Hermes Agent on your WEC Instance ā with SQLite-backed persistent memory that survives a full reboot. Real install, model config, and a memory-survives-restart test.", "date_modified": "2026-06-25T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "self-hosting", "hermes", "agents", "memory" ] }, { "id": "https://wec.wiline.com/docs/tutorials/netbird-mesh-vpn-openclaw/", "content_html": "+![\"NetBird\"](\"https://wec.wiline.com/docs/assets/images/netbird-dark-0d2ba9c783792904def84b7e687b8f95.png\")
In Article 2 we put Caddy in front of\nOpenClaw for HTTPS, and in Article 3\nwe added a Telegram channel. The gateway works ā but Caddy is still listening on\n0.0.0.0, reachable by anything that can route to the box. This is the capstone\nof the series: we join the server and your laptop to a NetBird mesh,\nrepoint openclaw.local at the mesh IP, and close the public ports. The same\nhttps://openclaw.local/chat URL keeps working ā but only for your devices.\nEvery command and error below is from the actual run.
Continues from Articles 1ā3 on the same WEC Instance ā Ubuntu, OpenClaw behind\nCaddy 2. We use managed NetBird (the free tier at\napp.netbird.io) with agent 0.73.2 on both the\nserver (Linux, kernel WireGuard) and the client (macOS, userspace). NetBird is\nWireGuard-based and open source; you can also\nself-host the control plane, which is a bigger topic for another day.
NetBird has two parts, and the distinction is the whole point:
\nSo you're building a mesh where Peer A (your laptop) reaches Peer B (the OpenClaw box) by its private 100.x address ā and openclaw.local points there, with Caddy bound to that interface only. The public/LAN side has nothing listening.
The key idea for later: the mesh isn't limited to two peers. Authenticate once, and every new service ā Ollama, a VDI, WordPress ā joins as another peer with its own 100.x address, reachable privately with no public ports to open. You set up the auth model once; you add services forever.
The public internet only ever reaches NetBird's control plane ā and that just\nauthenticates peers, it never carries your app traffic. Your services aren't\n\"firewalled off\" so much as invisible: OpenClaw has no public IP, so there's\nsimply no route to it from the internet. The only way in is to be an\nauthenticated peer on the mesh.
\nIn this tutorial we wire up the first two peers ā your laptop and the\nOpenClaw box. The other services make the point: once auth is set up,\nOllama, a VDI, WordPress each just join the same mesh as another 100.x\npeer ā no public ports, no per-service exposure.
Every NetBird mesh has two roles: peers ā your machines, i.e. the OpenClaw box\nfrom Parts 1ā3 and your laptop ā and a control plane that authenticates those\npeers and issues their join keys. You already have the peer (your OpenClaw box);\nnow you need a control plane for it to join. Two ways to get one:
\nWith Option A you'll run two instances: your OpenClaw box (Parts 1ā3,\nunchanged) and a second, dedicated control-plane box (this template). That's the\ncorrect topology ā a control plane should be its own machine, never sharing with a\nworkload. With Option B there's only your one OpenClaw box. Either way you\nnever rebuild the OpenClaw box ā in Enroll your machines\nyou just install the NetBird agent on it and it joins the mesh as a peer.
The steps below set up Option A. WEC runs the install; you give it an email and a\npublic IP, and a few minutes later you have your own NetBird dashboard on\nhttps://<your-ip>.apps.wiline.cloud with a valid certificate.
The dashboard needs a public address. Under Networks ā Elastic IPs, allocate a\nnew IP or pick a free one, and note its value (ours: 108.60.112.165). It must be\nin the VPC and zone you'll deploy into. Full steps:\nConfigure Elastic IPs.
![\"An](\"https://wec.wiline.com/docs/assets/images/netbird-eip-available-b4ec56b4c4946226b7c9e199bd540f61.png\")
This new instance becomes your NetBird control plane ā a separate box from your\nOpenClaw instance, which stays untouched. Open Deploy ā Deploy a VM (full walkthrough:\nDeploy a Virtual Machine).\nThe choices that matter here:
\nLocal Subnet in Princeton-VPC), so the IP can attach afterward.![\"Template](\"https://wec.wiline.com/docs/assets/images/netbird-template-config-e03939727bf27b7f6ce4ceb4fc5acdf1.png\")
Finish the wizard and deploy.
\nThe IP isn't bound at deploy time ā you attach it once the VM is up. Go to\nNetworks ā Elastic IPs ā your IP ā āÆ ā Allocate, and select your new instance\n(netbird-tutorial). Only VMs in the same VPC appear in the list.
![\"Allocating](\"https://wec.wiline.com/docs/assets/images/netbird-eip-allocate-620e7b57084f55e9e8ca3323494cda6d.png\")
The instance shows Running in Compute ā Instances within a minute ā but the\ntemplate keeps working for a few minutes after that, pulling and starting the\nNetBird containers and bringing up the dashboard. Don't expect the URL to answer\nimmediately.
\nCheck from your laptop over HTTPS (not ping ā ICMP is firewalled off, so a\nfailed ping doesn't mean it's down):
curl -skS -m 10 -o /dev/null -w \"%{http_code}\\n\" https://<your-ip>.apps.wiline.cloud
000 / connection refused ā still provisioning; wait a minute and retry.301 then 200 ā it's up (HTTP redirects to HTTPS).To watch it directly, SSH in (ssh ubuntu@<your-ip>) and confirm the containers are\nrunning and the ports are bound:
cloud-init status
sudo docker ps
sudo ss -tlnp | grep -E ':(80|443)'
It's ready once netbird-dashboard, netbird-traefik, and netbird-server show\nUp and 80/443 are listening.
Open https://<your-ip>.apps.wiline.cloud ā click through the certificate\nwarning if you get one (same self-signed-cert click-through as\nArticle 2).
This is your own NetBird control plane, running on your WEC Instance. On the\nfirst visit it asks you to create the admin account ā enter a name, email,\nand password, click Create, then sign in with those same credentials.\n(That setup screen appears only once; afterwards the dashboard shows a normal\nSign in page.)
\nYou land on the NetBird dashboard ā your private control plane on your own domain:
\n![\"The](\"https://wec.wiline.com/docs/assets/images/netbird-dashboard-c98eca329c445df75fee98ca55f230e9.png\")
Click Add Peer and NetBird hands you the enrollment command for your server ā\nnote the --management-url pointing at your own domain:
![\"Add](\"https://wec.wiline.com/docs/assets/images/netbird-add-peer-09fd78007bcc0279cae91217c1786221.png\")
curl -fsSL https://pkgs.netbird.io/install.sh | sh
netbird up --management-url https://<your-ip>.apps.wiline.cloud
(That second command opens a browser SSO login. For a headless server, add\n--setup-key <KEY> from your dashboard's Setup Keys page instead ā same as the\nmanaged flow, just with --management-url added.)
This is the dashboard where you create the setup keys that enroll machines ā which\nis exactly what we do next.
\nYou have a control plane (your own template dashboard, or managed NetBird). Now put\nthe two machines on the mesh ā the OpenClaw box from Parts 1ā3 and your\nlaptop ā and then close the box's public ports. These steps are the same no\nmatter which control plane you picked.
\nThe commands below use managed NetBird (app.netbird.io) as the example. If you\ndeployed the WEC template instead, it's identical except: create the setup key\nin your own dashboard (https://<your-ip>.apps.wiline.cloud) and add\n--management-url https://<your-ip>.apps.wiline.cloud to netbird up, so the\nagent points at your server instead of the managed cloud.
Prerequisites: OpenClaw running behind Caddy from\nArticles 1ā3 (reachable at\nhttps://openclaw.local), a control plane from the step above, and shell access to\nthe box plus admin on your laptop.
This is the box from Parts 1ā3 (a WEC Instance) ā we're enrolling it as a peer,\nwhich is separate from any control-plane server. NetBird ships a one-line installer:
\ncurl -fsSL https://pkgs.netbird.io/install.sh | sh
Gotcha (real one): on Ubuntu the install triggers the needrestart dialog ā\na blue \"Which services should be restarted?\" box. Tab between buttons can get\nswallowed by some SSH/terminal setups; use arrow keys to move, Space to\ntoggle, Enter to confirm (<Ok> is the default). The package is already in\nplace by the time this box appears, so even if you Ctrl+C out of it, the agent\nis installed ā re-running the installer just tells you so:
NetBird seems to be installed already, please remove it before proceeding
(If you re-run the installer after this tutorial, you'll instead see\nNetBird service is running, please stop it before proceeding ā the agent's\nalready up, so there's nothing to reinstall.)
Confirm the version:
\nnetbird version
# 0.73.2
A setup key is the token that enrolls a device into your mesh ā ideal for a\nheadless server (no browser SSO needed).
\nA few things will try to send you the wrong way on a fresh account ā here's the\nstraight line:
https://app.netbird.io/setup-keys.https://app.netbird.io/setup-keys.openclaw-tutorial, toggle Reusable (so it works for the laptop too), defaults otherwise.![\"Creating](\"https://wec.wiline.com/docs/assets/images/netbird-setup-key-ade9af748bded2998a18890a1ebde93b.png\")
Connect the VM (replace with your key):
\nsudo netbird up --setup-key <YOUR_SETUP_KEY>
Anyone with a reusable key can add devices to your network. Don't paste it into\nchats or commits, and delete it from the dashboard once your devices are\nenrolled (the existing peers stay connected).
Check the connection:
\nnetbird status
![\"netbird](\"https://wec.wiline.com/docs/assets/images/netbird-vm-status-8e8088e406d7392203f1eb5da63b4258.png\")
That NetBird IP (100.87.239.229) is the address everything below points\nat. Peers count: 0/0 is expected ā nothing else has joined yet.
Install the NetBird client on whatever you're using, then connect with the\nsame reusable setup key. The connect command (netbird up --setup-key ā¦) is\nidentical on every OS ā only the install differs.
The captured output below is from a Mac. The Windows and Linux tabs give the\nequivalent install; the netbird up, netbird status, and ping steps are the\nsame everywhere (Windows uses ping -n instead of -c).
Install via Homebrew, then start the service and connect:
brew install netbirdio/tap/netbird
sudo netbird service install
sudo netbird service start
netbird up --setup-key <YOUR_SETUP_KEY>
You may see a one-off gRPC warning as the daemon's socket comes up a beat after\nthe client calls it ā it ends in Connected, so it's harmless:
WARNING: ... dial unix /var/run/netbird.sock: connect: no such file or directory
Connected
Download and run the NetBird Windows installer from\napp.netbird.io (Add Peer ā Windows) or\npkgs.netbird.io. It installs a system-tray app and the\nnetbird CLI. Then, in PowerShell as Administrator, connect with your key:
netbird up --setup-key <YOUR_SETUP_KEY>
(You can also click Connect from the tray icon and authenticate via SSO, but\nthe setup key is the same one-shot flow as the other platforms.)
Same one-line installer as the VM, then connect:
curl -fsSL https://pkgs.netbird.io/install.sh | sh
sudo netbird up --setup-key <YOUR_SETUP_KEY>
Verify the two nodes see each other, and that the tunnel carries traffic:
\nnetbird status
ping -c 3 100.87.239.229 # Windows: ping -n 3 100.87.239.229
![\"Mac](\"https://wec.wiline.com/docs/assets/images/netbird-mac-status-857d291924930e23af4e6550b1845ec7.png\")
Peers count: 1/1 ā that peer is the VM. In the dashboard, both machines now\nshow up under Peers (because we enrolled them with a setup key rather than an\nSSO login, they appear under Servers rather than User Devices):
![\"The](\"https://wec.wiline.com/docs/assets/images/netbird-peers-a54440db77e8715febacc8d410c24da5.png\")
This path is going through a NetBird relay (note Interface type: Userspace\non the Mac). It's fine for a web UI, and NetBird will often upgrade to a direct\npeer-to-peer link when the networks allow. Latency to a relayed peer is not a\nred flag.
openclaw.local at the mesh IPāFrom Article 2, your hosts file maps\nopenclaw.local to the VM's LAN IP (10.80.4.212). Swap it for the mesh IP\n(100.87.239.229). The file path and edit command differ by OS:
sed on macOS needs the empty -i '':
grep openclaw /etc/hosts
# 10.80.4.212 openclaw.local
sudo sed -i '' 's/^10\\.80\\.4\\.212 openclaw\\.local/100.87.239.229 openclaw.local/' /etc/hosts
grep openclaw /etc/hosts
# 100.87.239.229 openclaw.local
GNU sed takes -i with no argument:
grep openclaw /etc/hosts
sudo sed -i 's/^10\\.80\\.4\\.212 openclaw\\.local/100.87.239.229 openclaw.local/' /etc/hosts
grep openclaw /etc/hosts
# 100.87.239.229 openclaw.local
The hosts file is at C:\\Windows\\System32\\drivers\\etc\\hosts and needs admin.\nIn PowerShell as Administrator:
$h = \"$env:WINDIR\\System32\\drivers\\etc\\hosts\"
(Get-Content $h) -replace '^10\\.80\\.4\\.212 openclaw\\.local','100.87.239.229 openclaw.local' | Set-Content $h
Select-String openclaw $h
(Or open that file in Notepad run as Administrator and change the IP by hand.)
Test over the mesh ā before touching any ports:
\ncurl -kI https://openclaw.local/
HTTP/2 200
...
via: 1.1 Caddy
via: 1.1 Caddy over the mesh IP ā the chat now flows through NetBird. Load\nhttps://openclaw.local/chat?session=main in the browser to confirm it behaves\nexactly as before. The TLS cert still matches because the hostname is unchanged;\nonly the IP behind it moved.
![\"The](\"https://wec.wiline.com/docs/assets/images/netbird-chat-mesh-409321edd81ea8d59656891246bc0558.png\")
Here's the state we're fixing. On the VM:
\nsudo ss -tlnp | grep -E ':(80|443)'
LISTEN 0 4096 0.0.0.0:443 ... docker-proxy
LISTEN 0 4096 0.0.0.0:80 ... docker-proxy
Caddy listens on 0.0.0.0 ā every interface, including the LAN (and any NAT\nin front of it).
Your first instinct is ufw deny 443. It won't work. Caddy's ports are\npublished by Docker, which writes its own iptables rules that bypass ufw's\nINPUT chain. You'd add ufw rules, feel safe, and 443 would stay wide open. On\nthis box ufw was even inactive ā and enabling it would have changed nothing\nfor the Docker-published ports.
The reliable fix is to bind the published ports to the mesh IP instead of\n0.0.0.0, so Caddy only ever listens on wt0. Edit the Caddy service in\ndocker-compose.override.yml (the override from Article 2):
cd /home/ubuntu/openclaw
cp docker-compose.override.yml docker-compose.override.yml.bak
Change the ports block from:
\n ports:
- \"80:80\"
- \"443:443\"
to bind to the mesh IP:
\n ports:
- \"100.87.239.229:80:80\"
- \"100.87.239.229:443:443\"
Recreate just Caddy and re-check:
\ndocker compose up -d caddy
sudo ss -tlnp | grep -E ':(80|443)'
![\"ss](\"https://wec.wiline.com/docs/assets/images/netbird-ss-after-0172e6fc62bc667a141b1f6528526e73.png\")
The ports are gone from 0.0.0.0 ā they live on the mesh interface only.
Because the bind targets the mesh IP, NetBird must be up before Caddy starts\n(e.g. after a reboot, wt0 needs its IP before Docker binds to it). The\nrestart: unless-stopped policy already covers this ā Caddy retries until the\ninterface exists ā but it's worth knowing if you ever see Caddy crash-looping\nright after boot.
Both directions, from the actual run.
\nStill reachable over the mesh (laptop):
\ncurl -kI https://openclaw.local/
# HTTP/2 200 ... via: 1.1 Caddy
No longer reachable on the LAN (VM, hitting its own LAN IP):
\ncurl -kI --connect-timeout 5 --resolve openclaw.local:443:10.80.4.212 https://openclaw.local/
![\"On](\"https://wec.wiline.com/docs/assets/images/netbird-refused-0944b25666489470ae7b961e2d6243c8.png\")
That's the result: same URL, reachable only through your mesh, refused\neverywhere else.
\nYou can now take any self-hosted service off the public internet without breaking how you\nuse it.
Delete the reusable setup key now that both devices are enrolled ā\nSetup Keys ā delete in the dashboard. Your peers stay connected; you're just\nclosing the door you used to add them. To remove a device later, delete it under\nPeers ā User Devices and run netbird down on that machine.
needrestart dialog won't take Tab. Use arrow keys / Space / Enter. The package is already installed by the time the box shows.netbird.sock warning on first up. The daemon's socket lags the client by a beat. If it ends in Connected, ignore it.ufw rules don't block the gateway. Docker-published ports bypass ufw ā bind the port to a specific IP (as above) or use Docker's own DOCKER-USER chain.wt0 had the mesh IP. restart: unless-stopped recovers it; or order Docker after netbird.service.Interface type and NetBird's peer detail for the connection type.That closes the Self-hosting OpenClaw series:
\nYou now have a self-hosted agent that's encrypted in transit, paired to your\ndevices, reachable from your phone, and invisible to the public internet ā all on\none small WEC Instance.
\nReady for an agent that remembers? Start the Self-hosting Hermes series ā\nSelf-host the Hermes Agent with persistent memory ā\ndeployed on this same WEC Instance, with context that survives restarts.
", "url": "https://wec.wiline.com/docs/tutorials/netbird-mesh-vpn-openclaw/", "title": "Make OpenClaw private with a NetBird mesh VPN", "summary": "Put OpenClaw on a private mesh with NetBird, repoint your hostname at the mesh IP, and close the public ports ā so the same chat URL works only for your devices. Real commands and gotchas from a live run.", "date_modified": "2026-06-24T00:00:00.000Z", "author": { "name": "Rafael Fernandes", "url": "https://www.linkedin.com/in/rafaelmacariofernandes/" }, "tags": [ "ai", "self-hosting", "openclaw", "netbird", "vpn", "mesh", "security" ] }, { "id": "https://wec.wiline.com/docs/tutorials/add-telegram-channel-openclaw/", "content_html": "+You've deployed OpenClaw and\nsecured it behind HTTPS. Now make it\nusable ā talk to your agent from your phone via Telegram. We create a bot,\nconnect it, clear OpenClaw's pairing gate, and get a real reply. Every command and\ngotcha below is from an actual run.
\n\nContinues from Parts 1ā2 on the same WEC Instance ā OpenClaw 2026.6.8. Telegram\nuses polling (the gateway polls Telegram's API), so no public webhook or domain\nis required ā this works on a private box.
Your phone talks to a Telegram bot; the OpenClaw gateway polls Telegram for\nmessages, runs them through your model, and replies ā no inbound ports needed.
\n\n~/openclawweb.telegram.org. (You can't create a bot from the public t.me web page; see Troubleshooting.)In the Telegram app, tap search and type BotFather, then open the official\naccount ā the one with the blue checkmark.
![\"Searching](\"https://wec.wiline.com/docs/assets/images/tg-step1a-search-922bcf2cacaf250ac10c1a03546f240b.png\")
Tap Start ā BotFather replies with the commands it understands (you'll see\noptions like /newbot, /mybots, /token, and more). Send /newbot, then\nfollow its two prompts:
My OpenClaw Agentbot, e.g. wiline_openclaw_botBotFather confirms with \"Done! Congratulations on your new bot\" and gives you the\ntoken ā the line after Use this token to access the HTTP API:, like\n1234567890:AA....
![\"BotFather](\"https://wec.wiline.com/docs/assets/images/tg-step1c-token-0bec47ec554ad66500f6d053bd0a4810.png\")
Anyone with it controls your bot. Blur it in the screenshot above, and /revoke\nit in BotFather if it ever leaks.
On the VM, in ~/openclaw, register the channel with your token:
docker compose run --rm openclaw-cli channels add --channel telegram --token \"<your-bot-token>\"
# Added Telegram account \"default\".
Restart the gateway so it picks up the channel, then check status:
\ndocker compose restart openclaw-gateway
docker compose run --rm openclaw-cli channels status
# - Telegram default: enabled, configured, running, mode:polling, token:config
mode:polling means the gateway is actively polling Telegram ā no inbound port or\nwebhook required.
![\"channels](\"https://wec.wiline.com/docs/assets/images/tg-step3-status-6393f1690ec9b270312c2f5e3a25f137.png\")
Open your bot (t.me/<your-bot-username>), tap Start, and send a message.
OpenClaw treats unknown senders as untrusted ā so instead of replying, the bot\nreturns a pairing code and your Telegram user id. This is the same security\npattern as device pairing in Part 2.
\n![\"Bot](\"https://wec.wiline.com/docs/assets/images/tg-step4-pairing-94e777f1790fbeadab0cd42be384b80b.png\")
On the VM, approve the code from the bot's message:
\ndocker compose run --rm openclaw-cli pairing approve telegram <pairing-code>
# Approved telegram sender <id>.
# Command owner configured telegram:<id> (commands.ownerAllowFrom was empty).
Approving the first user as command owner changes config that requires a\ngateway restart ā OpenClaw does this automatically (you'll see\nconfig change requires gateway restart then the Telegram provider restart in the\nlogs). Wait a few seconds for it to come back before messaging again.
Back in Telegram, just talk to it ā no extra setup needed; OpenClaw with your model\nanswers conversationally out of the box. Try something like:
\nIn one sentence, what's the point of putting a reverse proxy in front of my agent?
It replies with a real answer ā the full chain is live: your phone ā Telegram ā\ngateway ā OpenAI ā reply. Your self-hosted agent is now in your pocket. (Giving it\ntools and skills comes in later guides; conversational chat works immediately.)
\n![\"The](\"https://wec.wiline.com/docs/assets/images/tg-step6-working-7acac4effbd1c03d0657534125e3ea3b.png\")
Your agent is now in your pocket ā a private Telegram line to your own AI, no public\nwebhook needed.
Opening t.me/BotFather in a plain web browser shows a Start button that does\nnothing ā that page is just the public bot card and needs a logged-in Telegram\nclient to hand off to. Use the desktop/phone app, or log in at\nweb.telegram.org (link a desktop device by scanning the QR from your phone:\nSettings ā Devices ā Link Desktop Device), then message BotFather there.
Not an error ā it's the pairing gate. Unknown senders must be approved first\n(Step 5). After approval, the bot chats normally.
\nProxy headers detected from untrusted addressāIf you followed Part 2, the gateway logs this because Caddy sits in front of it.\nIt's a warning, not a failure. To silence it and restore correct client\ndetection, set gateway.trustedProxies to your proxy's network.
event loop degraded under load.\nFor a busy bot, give it more CPU/RAM.You've finished Part 3 of the Self-hosting OpenClaw series. Next:
\nThen start the Self-hosting Hermes series ā Self-host the Hermes Agent with persistent memory ā for an agent whose context survives restarts.
\nCompanion files for the series live in the WiLine manifests repo (link coming with the repo).
\nRemove the Telegram channel (keeps the rest of OpenClaw running):
\ndocker compose run --rm openclaw-cli channels remove --channel telegram
docker compose restart openclaw-gateway
And /revoke the bot token in BotFather if you're done with it.
+In Article 1 we got OpenClaw running ā\nbut only over plain HTTP, with an allowInsecureAuth workaround. Here we put\nCaddy in front of it as a reverse proxy: real HTTPS,\ndevice-paired auth, and the gateway's raw ports closed so the proxy is the only\nway in. Every command and error below is from the actual deploy.
Continues from Article 1 on the same WEC Instance ā Ubuntu 22.04.5 LTS, OpenClaw\n2026.6.8. We use Caddy 2 (caddy:2). Because this box has no public domain,\nwe use Caddy's internal CA (tls internal) for self-signed HTTPS; the\nGoing to production section shows the one-line swap to real\nLet's Encrypt certificates once you have a domain.
Caddy terminates HTTPS and reverse-proxies to the OpenClaw gateway over Docker's\ninternal network. The gateway stops publishing its own ports ā the only entry\npoint becomes Caddy on :443.
~/openclawopenclaw.local + a hosts entry.cd ~/openclaw
docker compose ps
You should see openclaw-gateway Up (healthy), publishing 18789/18790/3978.\nThat's our starting state: working, but HTTP-only.
Caddy's config: serve HTTPS and proxy everything to the gateway. tls internal\nuses Caddy's own local CA ā no public domain needed.
cat > Caddyfile <<'EOF'
# Caddy selects its certificate by SNI (the hostname in the TLS handshake).
# A bare IP sends no SNI, so we use a hostname. With a public domain, put your
# domain here and drop `tls internal` for automatic Let's Encrypt.
openclaw.local {
tls internal
reverse_proxy openclaw-gateway:18789
}
EOF
The proxy target is openclaw-gateway:18789 ā Caddy reaches the gateway by its\nCompose service name over the shared Docker network.
docker compose auto-merges a docker-compose.override.yml, so we add a caddy\nservice without touching OpenClaw's official compose file:
cat > docker-compose.override.yml <<'EOF'
services:
caddy:
image: caddy:2
restart: unless-stopped
ports:
- \"80:80\"
- \"443:443\"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config
volumes:
caddy_data:
caddy_config:
EOF
caddy_data persists the local CA + certs across restarts.
docker compose up -d
docker compose ps
Caddy comes up alongside the gateway with 443 published, and generates its\ninternal CA + a certificate on first run.
![\"Caddy](\"https://wec.wiline.com/docs/assets/images/caddy-step4-up-d53a3b63c4e346428a8e733249b2049a.png\")
docker compose up -d also starts the one-shot openclaw-cli helper; it idling\nhere is harmless ā we invoke it with docker compose run --rm when needed.
Hit the gateway's health endpoint through Caddy. --resolve makes curl send\nthe hostname (SNI) while pointing at the box:
curl -k --resolve openclaw.local:443:127.0.0.1 https://openclaw.local/healthz
# {\"ok\":true,\"status\":\"live\"}
tlsv1 alert internal error?That happens if you used a bare IP as the site address ā see\nTroubleshooting #1. Use a hostname.
Now traffic arrives over https://openclaw.local. Two config changes:
allowedOrigins (the Control UI rejects unknown origins).allowInsecureAuth back off ā HTTPS is a secure context, so the\nArticle 1 workaround is no longer needed.docker compose run --rm --no-deps --entrypoint node openclaw-gateway \\
dist/index.js config set --batch-json '[{\"path\":\"gateway.controlUi.allowedOrigins\",\"value\":[\"https://openclaw.local\"]},{\"path\":\"gateway.controlUi.allowInsecureAuth\",\"value\":false}]'
docker compose restart openclaw-gateway
Open https://openclaw.local/ in your browser (on a private box, first map\n10.80.4.212 openclaw.local in your machine's hosts file ā /etc/hosts on\nmacOS/Linux, or C:\\Windows\\System32\\drivers\\etc\\hosts edited as Administrator\non Windows).
You'll hit a certificate warning first ā \"Your connection is not private\"\n(NET::ERR_CERT_AUTHORITY_INVALID). This is expected with tls internal: the\ncertificate is signed by Caddy's local CA, which your operating system doesn't\ntrust. Click Advanced ā Proceed to openclaw.local (unsafe) to continue. (With a\nreal domain + Let's Encrypt, the certificate is publicly trusted and this warning\nnever appears ā see Going to production.)
![\"Browser](\"https://wec.wiline.com/docs/assets/images/caddy-step7-cert-warning-f6015d4f4498bdd7f495c01e08e91cf5.png\")
Now the page loads ā but with allowInsecureAuth off, the gateway requires\ndevice pairing before this browser can use the Control UI. That's the secure\nflow doing its job. The screen shows a request id; approve it on the host:
docker compose run --rm openclaw-cli devices approve <request-id-from-the-screen>
![\"Device](\"https://wec.wiline.com/docs/assets/images/caddy-step7-pairing-95202988ec2160fdef5dcd2e33fce67f.png\")
Back in the browser, click Connect again ā it pairs and loads the dashboard.
\nThe gateway still publishes 18789 to the host, so http://<vm-ip>:18789 is\nstill reachable in plain HTTP, bypassing Caddy. Close it: stop publishing the\ngateway's ports so Caddy on :443 is the only door. Caddy still reaches it over\nthe internal network.
cat > docker-compose.override.yml <<'EOF'
services:
openclaw-gateway:
# Unpublish the gateway's host ports ā Caddy reaches it internally, so the
# only entry point is HTTPS on :443.
ports: !reset []
caddy:
image: caddy:2
restart: unless-stopped
ports:
- \"80:80\"
- \"443:443\"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config
volumes:
caddy_data:
caddy_config:
EOF
docker compose up -d
ports: !reset [] clears the ports the base compose published.
curl -s -m 5 http://<vm-ip>:18789/healthz ; echo \" <- direct HTTP (should FAIL)\"
curl -sk --resolve openclaw.local:443:127.0.0.1 https://openclaw.local/healthz ; echo \" <- via Caddy (should work)\"
The first returns nothing (connection refused) ā the insecure door is gone. The\nsecond still returns {\"ok\":true,\"status\":\"live\"}.
Your OpenClaw is now reachable only over HTTPS, device-paired.
\n![\"OpenClaw](\"https://wec.wiline.com/docs/assets/images/caddy-control-ui-https-814c25fc8dd38568d2226856190421ba.png\")
You can now put real HTTPS and auth in front of any self-hosted service ā and prove the\nback door is closed.
tlsv1 alert internal errorāHit when the Caddyfile site address was a bare IP (https://10.80.4.212):
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error
Caddy's logs showed the cert was obtained fine. Cause: TLS picks a certificate\nby SNI (the hostname in the handshake), but connections to a bare IP send no\nSNI ā so Caddy can't match a cert and aborts. Fix: use a hostname site address\n(Step 2). A reader with a real domain never hits this ā the domain is the SNI.
\nOn the first HTTPS connect after turning off allowInsecureAuth:
Device pairing required ā This browser needs one-time approval from the Gateway
host before it can use the Control UI.
Cause: this is the correct secure flow ā with insecure auth off, each new\nbrowser must be approved. Fix: docker compose run --rm openclaw-cli devices approve <request-id>. (The CLI may log scope upgrade pending approval ā¦ using local fallback but still completes the approval.)
NET::ERR_CERT_AUTHORITY_INVALIDāExpected with tls internal ā the cert is signed by Caddy's local CA, which your\nOS doesn't trust. Click through on a private box. With a public domain + Let's\nEncrypt (below), the cert is trusted and there's no warning.
With a real domain, swap two lines in the Caddyfile ā point it at your domain and\ndrop tls internal:
- openclaw.local {
- tls internal
+ openclaw.example.com {
reverse_proxy openclaw-gateway:18789
}
Then point the domain's DNS A record at the VM's public IP and open 80 + 443\nto the internet. Caddy automatically provisions and renews a Let's Encrypt\ncertificate ā trusted, no warning, no --resolve or /etc/hosts needed. Update\nallowedOrigins to https://openclaw.example.com.
:443 is exposed; the gateway has no host\nports (Step 8).openclaw devices).https://openclaw.local.OPENCLAW_GATEWAY_TOKEN if it has ever been\nshown (screenshots, screen-shares).You've finished Part 2 of the Self-hosting OpenClaw series. Next:
\nCompanion files (Caddyfile, docker-compose.override.yml) live in the WiLine\nmanifests repo (link coming with the repo).
To remove just Caddy and reopen the direct ports, delete docker-compose.override.yml\nand Caddyfile, then docker compose up -d. To stop everything:
docker compose down
Your AI assistant doesn't have to live in someone else's cloud.
\nDeploy a self-hosted OpenClaw AI agent on a WEC Instance with\nDocker Compose ā from spinning up the VM to an agent that actually answers, using\nyour own model API key. Every command, version, and error below was captured\nfrom a real deployment on a WEC Instance.
\n\nYou bring your own model API key (OpenAI in this guide); WiLine handles the\nhosting. This was reproduced on a WEC Instance ā Ubuntu 22.04.5 LTS, 8 vCPU,\n15 GiB RAM, 25 GB free disk. OpenClaw version 2026.6.8, image\nghcr.io/openclaw/openclaw:latest.
A single OpenClaw gateway container, configured with your model provider,\nreachable through its web Control UI and CLI.
\n\n~/.ssh/id_ed25519 or similar)Spin up a WiLine VM that meets these requirements:
\n| Requirement | Value |
|---|---|
| OS | Ubuntu 22.04 LTS (or 24.04 LTS) |
| Compute | ā„ 2 vCPU / 4 GB RAM ā OpenClaw needs ā„ 2 GB; the first run can be OOM-killed (exit 137) on 1 GB hosts |
| Disk | 30 GB NVMe ā 10 GB fills up fast once Docker images and logs land |
| Access | Your SSH public key added at deploy time, so you can log in without a password |
The portal walkthrough is already documented ā follow these and come back:
\n\nOnce it's running, SSH in:
\nssh ubuntu@<your-vm-ip>
Before installing anything, confirm what you're working with ā the OS version and\nthe resources available. Note these numbers; they're useful when you later compare\nperformance or open a support ticket.
\nlsb_release -a
nproc
free -h
df -h /
On our WEC Instance: Ubuntu 22.04.5 LTS, 8 vCPU, ~15 GiB RAM, 25 GB free disk.
![\"Terminal](\"https://wec.wiline.com/docs/assets/images/step2-box-specs-c821280774bf4f5ec9205092f6646004.png\")
Check whether Docker is already present:
\ndocker --version
docker compose version
If it's not installed, follow Docker's official guide for your distro ā\nInstall Docker Engine on Ubuntu.\nThe quickest path is Docker's\nconvenience script,\nfollowed by adding your user to the docker group so you can run it without\nsudo:
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
You need Docker Compose v2 (the docker compose subcommand, not the legacy\ndocker-compose binary).
If Docker is already present, the get.docker.com script detects it and warns\nrather than reinstalling ā don't force it on a box with running containers. On a\nclean VM it installs from scratch. This deploy ran on a host where Docker\n29.1.3 / Compose v5.0.2 was already installed.
Create a project directory and pull OpenClaw's official docker-compose.yml:
mkdir openclaw && cd openclaw
curl -fsSL https://raw.githubusercontent.com/openclaw/openclaw/main/docker-compose.yml -o docker-compose.yml
The file defines two services ā openclaw-gateway (the long-running agent\ngateway) and openclaw-cli (a one-shot CLI that shares the gateway's network).\nBy default it builds from source; we'll point it at the prebuilt image\ninstead.
.envāCreate a .env next to the compose file. Pin the prebuilt image and set a\ndashboard token:
OPENCLAW_IMAGE=ghcr.io/openclaw/openclaw:latest
OPENCLAW_GATEWAY_TOKEN=<paste output of: openssl rand -hex 32>
.env out of gitIt holds your gateway token (and any keys). Add it to .gitignore. Commit a\n.env.example with placeholder values instead.
Pull the image:
\ndocker compose pull
OpenClaw bind-mounts ~/.openclaw into the container. Create it as your own\nuser first ā otherwise Docker creates it as root and the container (which\nruns as user node, uid 1000) can't write to it:
mkdir -p ~/.openclaw ~/.openclaw-auth-profile-secrets
If you skip this, you'll hit a permission error on first run ā see\nTroubleshooting.
\nThis is where you bring your own model ā onboarding prompts for your provider and\nAPI key. This guide uses OpenAI, but Anthropic and local models work the same way.
\nRun OpenClaw's onboarding through the gateway container. It's interactive ā it\nasks for your provider and API key and writes the config:
\ndocker compose run --rm --no-deps --entrypoint node openclaw-gateway \\
dist/index.js onboard --mode local --no-install-daemon
Walk through the prompts:
\nopenai/gpt-5.5)skip ā Skip for now (messaging channels are a later\nguide)openclaw configure)It ends with Onboarding complete.
Onboarding wrote your provider and model. Now set the gateway's runtime options ā\nrun mode, network bind, and the Control UI origins it will accept ā then start it\nas a background service.
\ndocker compose run --rm --no-deps --entrypoint node openclaw-gateway \\
dist/index.js config set --batch-json '[{\"path\":\"gateway.mode\",\"value\":\"local\"},{\"path\":\"gateway.bind\",\"value\":\"lan\"},{\"path\":\"gateway.controlUi.allowedOrigins\",\"value\":[\"http://localhost:18789\",\"http://127.0.0.1:18789\"]}]'
docker compose up -d openclaw-gateway
Container health:
\ndocker compose ps
You want Up ... (healthy). Then hit the health endpoint:
curl http://127.0.0.1:18789/healthz
# {\"ok\":true,\"status\":\"live\"}
![\"docker](\"https://wec.wiline.com/docs/assets/images/step9-health-2a25a28bc6b53dd20c385259373d72a5.png\")
Confirm your provider key is usable:
\ndocker compose run --rm openclaw-cli models status
# openai ... api_key=1 ... status=usable
models status prints a masked key prefix (sk-proj-ā¦). Crop or blur it before\npublishing the screenshot.
Now the real end-to-end test. The agent command needs a target, so list agents\nfirst, then message the default one (main):
docker compose run --rm openclaw-cli agents list
# - main (default)
docker compose run --rm openclaw-cli agent --agent main \\
--message \"Reply with exactly one word: working\"
# working
If it replies working, the full chain ā gateway ā OpenAI ā response ā is live.
![\"The](\"https://wec.wiline.com/docs/assets/images/step9-agent-working-91514c6b42235714c1348d5ce9f6e9b9.png\")
The Control UI is at http://<your-instance-ip>:18789/; paste your\nOPENCLAW_GATEWAY_TOKEN into Settings to log in.
Browsers block the Control UI over plain HTTP at a remote IP (\"Secure browser\ncontext required\"). Reach it over an SSH tunnel so it loads as 127.0.0.1 ā see\nTroubleshooting #3.
![\"OpenClaw](\"https://wec.wiline.com/docs/assets/images/step9-control-ui-fe1df8b0971ad1a8192ff40d2ea6856e.png\")
These are errors we actually hit during this deploy ā not hypotheticals.
\nEACCES: permission denied, mkdir '/home/node/.openclaw/state'āHit on the first onboarding run when ~/.openclaw didn't pre-exist:
[openclaw] The CLI command failed.
[openclaw] Reason: Failed to open the plugin state database.
| EACCES: permission denied, mkdir '/home/node/.openclaw/state' | EACCES
Cause: Docker auto-created the bind-mount source ~/.openclaw owned by\nroot (0:0). The container runs as user node (uid 1000), so it can't write\ninside that root-owned directory.
Fix: give the directory to uid 1000 (which is also your ubuntu user):
sudo chown -R 1000:1000 ~/.openclaw
Pre-creating the directory yourself (Step 6) avoids this entirely.
\n![\"EACCES](\"https://wec.wiline.com/docs/assets/images/error1-eacces-fbbf355224203ecfccd68552fcf0bcb3.png\")
No target session selectedāThe first time we ran agent without specifying who to talk to:
Error: No target session selected. Use --agent <id>, --session-key <key>,
--session-id <id>, or --to <E.164>. Run openclaw agents list to see agents.
Cause: openclaw agent runs one turn against a specific agent/session; with\nno channel configured there's no implicit target.
Fix: list agents and pass the id explicitly:
\ndocker compose run --rm openclaw-cli agents list # shows: main (default)
docker compose run --rm openclaw-cli agent --agent main --message \"...\"
![\"The](\"https://wec.wiline.com/docs/assets/images/error2-no-session-c405ca89ca87bd411080165c514046e5.png\")
Opening the Control UI from another machine (e.g. macOS) at the VM's IP over\nplain HTTP, the page refuses to connect:
\nSecure browser context required
This page is running over plain HTTP, so the browser cannot create the
device identity the Gateway expects.
Cause: browsers only expose the crypto APIs OpenClaw needs in a secure\ncontext ā HTTPS, or localhost/127.0.0.1. A remote http://<ip>:18789 is\nneither, so it's blocked in the browser, regardless of the server-side\ngateway.controlUi.allowInsecureAuth flag.
Fix: tunnel the port over SSH so the browser talks to 127.0.0.1 (a secure\ncontext). On your local machine:
ssh -L 18789:127.0.0.1:18789 ubuntu@<your-vm-ip>
Then open http://127.0.0.1:18789/#token=<your-token> locally. The proper\nlong-term fix is HTTPS via a reverse proxy ā the next guide.
![\"The](\"https://wec.wiline.com/docs/assets/images/error3-secure-context-3e5792a56fae6a2ea2a8f9e0f0aa4c3f.png\")
On startup the gateway logged warnings worth acting on before any public\nexposure:
\ngateway.controlUi.allowInsecureAuth=true ā flagged dangerous. Run\ndocker compose run --rm openclaw-cli security audit.plugins.allow is empty ā non-bundled plugins (e.g. codex) may\nauto-load. Set an explicit allowlist of trusted plugin ids.Hardening this behind a reverse proxy with HTTPS is the subject of the next\nguide.
\nYou can now deploy and operate your own AI agent on a box you control.
You went from a bare WEC Instance to a working, self-hosted AI agent:
\nThe agent is yours: your key, your box, your data. From here it's about making it\nsecure, reachable, and useful.
\nOpenClaw isn't a chatbot ā it's an autonomous agent that can act on your box. Now\nthat it's running, here's what it unlocks (some features need extra configuration, and\nwe cover the big ones later in this series):
\nTypical real uses people run: inbox and calendar management, document processing,\ninfrastructure tasks on the box, content pipelines, and autonomous code testing.
\nThis is Part 1 of the Self-hosting OpenClaw series on running production-grade\nAI infrastructure on a WEC Instance. Coming up:
\nallowInsecureAuth workaround and reach the Control UI from anywhere, safely.Companion files for this guide ā docker-compose.yml and .env.example ā live in\nthe WiLine manifests repo (link coming with the repo).
docker compose down # stop + remove containers
docker compose down --rmi all -v # also remove the image and volumes
To remove config/state on the host:\nrm -rf ~/.openclaw ~/.openclaw-auth-profile-secrets. To stop billing entirely,\ndelete the VM from the WiLine portal.