LiteLLM complexity routing: the right model for each request, and what it costs in latency
+
+- 1One endpoint, scoped keys
- 2Route work to the right model
- 3Mask PII at the gateway
- 4Clean traces, untouched answers
- 🏆Prove it holds under load
Part 1 ended on an uncomfortable number. The same three-word answer cost 3 tokens from a small model and 200 from a reasoning model — which spent all 200 thinking and returned nothing at all.
Every request your apps send picks a model, and mostly that choice is made once, hardcoded, and never revisited. This post puts the gateway in charge of it instead: classify the request, route it to a model sized for the work. Then it measures what that decision costs, because it is not free and most write-ups skip that part.
Adding LiteLLM's complexity router
The gateway from Part 1 already has three models registered. A router is just
another entry in model_list that maps complexity tiers onto them:
- model_name: smart-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: qwen-small
MEDIUM: qwen-mid
COMPLEX: qwen-large
REASONING: qwen-large
return_raw_model_name: true
return_raw_model_name: true is the important one for now. Without it the
response reports smart-router and you have no idea which model served you.
With it, the response names the model that actually ran — so every test below
is self-verifying.
docker compose restart litellm
Watching it route
You'll send the same request many times with only the prompt changing, so wrap it once:
ask() { curl -s http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d "{\"model\":\"smart-router\",\"messages\":[{\"role\":\"user\",\"content\":\"$1\"}],\"max_tokens\":10}" | jq -r .model; }
Then five prompts of increasing difficulty:
ask "hi"
ask "what is a vpc"
ask "write a python function that retries an http call with backoff"
ask "our two services deadlock under load, walk me through diagnosing it"
ask "prove that the halting problem is undecidable"
One line per prompt, in the order sent:
Qwen2.5-3B-Instruct
Qwen2.5-3B-Instruct
Qwen3.5-9B
Qwen2.5-3B-Instruct
Qwen2.5-3B-Instruct
Figure 1. The code request went up a tier. The two hardest questions did not.
A greeting on the small model is right. A Python request on the mid model is right. But a distributed-systems deadlock and one of the hardest questions in computer science both landed on a 3-billion-parameter model.
To understand why, you have to know what the router is actually doing.
What a heuristic is, and how LiteLLM scores one
By default this router makes zero API calls. It scores the prompt locally with pattern matching — that's what "heuristic" means here: a cheap rule of thumb that approximates a judgement without making it.
It scores seven dimensions, each producing a value between −1 and +1, then multiplies each by a fixed weight and adds them up:
| Dimension | Weight | Fires on |
|---|---|---|
codePresence | 0.30 | function, class, api, schema, … |
reasoningMarkers | 0.25 | "step by step", "think through", "analyze" |
technicalTerms | 0.25 | "architecture", "distributed", "encryption" |
tokenCount | 0.10 | −1.0 under 15 tokens, +1.0 over 400 |
simpleIndicators | 0.05 | "what is", "define", greetings — scores −1.0 |
multiStepPatterns | 0.03 | "first… then", numbered steps |
questionComplexity | 0.02 | more than three question marks |
The weighted sum maps to a tier at three boundaries: below 0.15 is SIMPLE, below 0.35 MEDIUM, below 0.60 COMPLEX, and above that REASONING. Those values are all documented and all configurable.
Two dimensions can push the score down. A short prompt scores −1.0 on
tokenCount. A prompt containing "what is" scores −1.0 on simpleIndicators.
The arithmetic, on a real prompt
The router records its own working. Every routed request writes a decision into the spend log:
curl -s http://127.0.0.1:4000/spend/logs \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
| jq -r '.[].metadata.routing_decision | select(.) | "\(.tier) \(.score) -> \(.routed_model) \(.signals)"' | head -5
Newest first, so this reads bottom-up against the order the prompts were sent:
SIMPLE -0.1 -> qwen-small ["short (11 tokens)"]
SIMPLE 0 -> qwen-small null
MEDIUM 0.3 -> qwen-mid ["code (function, python)"]
SIMPLE -0.15000000000000002 -> qwen-small ["short (3 tokens)","simple (what is)"]
SIMPLE -0.15000000000000002 -> qwen-small ["short (0 tokens)","simple (hi)"]
Figure 2. Not a black box — the router reports the score and the signals it fired.
Take the last one. prove that the halting problem is undecidable:
tokenCount -1.0 × 0.10 = -0.10 11 tokens, under the 15 threshold
codePresence 0.0 × 0.30 = 0.00 no code keywords
reasoningMarkers 0.0 × 0.25 = 0.00 "prove" is not in the marker list
technicalTerms 0.0 × 0.25 = 0.00
simpleIndicators 0.0 × 0.05 = 0.00
multiStepPatterns 0.0 × 0.03 = 0.00
questionComplexity 0.0 × 0.02 = 0.00
------
-0.10 below 0.15 → SIMPLE → 3B model
Every term is zero except a penalty for being short. The question gets routed down because it is brief.
Now the deadlock prompt, which is worse: it scored 0.00 with no signals at all. Not a low score — nothing matched. "Our two services deadlock under load, walk me through diagnosing it" contains no keyword the scorer recognises, so it falls to SIMPLE by default.
You can read a routing decision — tier, score, and the signals that produced it — and reproduce the arithmetic by hand from the dimension weights.
The router isn't malfunctioning. It is doing exactly what its rules say. The rules just have no way to see difficulty that isn't spelled out in vocabulary it knows.
The failure mode is systematic, not random: short prompts that need deep thinking get routed down. Those are also the prompts where a wrong model is most obvious to the user.
A keyword never matches its own plural
Single-word keywords are matched on word boundaries, so endpoint does not match
endpoints. Every single-word keyword in the default lists is singular, and none
of them match a plural.
Whether that changes anything depends on how close the score already sits to a
boundary, because the dimensions are stepped rather than linear — technicalTerms
scores 0.5 at two matches and 1.0 at four, so losing one match often changes
nothing. Sometimes it changes the model:
Review our api endpoint for authentication and authorization problems → COMPLEX +0.425
Review our api endpoints for authentication and authorization problems → MEDIUM +0.275
One letter, one tier. The direction is always the same: a plural scores lower or equal, never higher, so the drift is toward the cheaper model.
Plurals you care about can be appended to the technical list with
custom_technical_keywords. There is no equivalent for code keywords — the only
lever is code_keywords, which replaces the built-in list rather than extending
it.
Asking a model instead
The alternative is to spend a model call on the decision. Four lines:
classifier_type: llm
classifier_llm_config:
model: qwen-small
timeout_ms: 3000
Now the router sends the prompt to qwen-small with a rubric and a schema that
forces back exactly one of SIMPLE, MEDIUM, COMPLEX, REASONING, and routes
on the answer. The classifier here is the cheapest model we have — the same 3B
that was wrongly answering the hard questions a moment ago. It turns out to be a
better judge of difficulty than it is an answerer of it.
Restart, then re-run the identical five prompts:
Qwen2.5-3B-Instruct
Qwen3.5-9B
Qwen3.5-122B
Qwen3.5-122B
Qwen3.5-122B
Figure 3. Both failures corrected — and two prompts moved up that arguably shouldn't have.
| Prompt | Heuristic | LLM classifier |
|---|---|---|
hi | 3B | 3B |
what is a vpc | 3B | 9B |
python function … backoff | 9B | 122B |
deadlock under load | 3B ✗ | 122B ✓ |
halting problem | 3B ✗ | 122B ✓ |
The two broken cases are fixed. But read rows 2 and 3 again — everything moved up. "What is a vpc" is a factual lookup a 3B model answers perfectly well, and it now runs on the 9B. You stopped under-serving hard prompts and started over-serving easy ones. Whether that trade is worth it depends on your traffic mix, and you should measure yours rather than trust this table.
What it costs
This is the part that gets left out. Add a twin that skips the router, then time both:
direct() { curl -s http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d "{\"model\":\"qwen-small\",\"messages\":[{\"role\":\"user\",\"content\":\"$1\"}],\"max_tokens\":10}" | jq -r .model; }
time ask "hi"
time ask "hi"
time direct "hi"
Qwen2.5-3B-Instruct real 0m2.474s ← first call after restart
Qwen2.5-3B-Instruct real 0m1.041s
qwen-small real 0m0.529s
Figure 4. Same prompt, same model answers it, twice the wall clock.
The same 3B model produced both answers. The only difference is that one request was classified first. Now decompose it — the spend log times each call separately:
| Wall clock | Classifier call | Serving call |
|---|---|---|
| 2.474 s (cold) | 1449 ms | 806 ms |
| 1.041 s | 549 ms | 456 ms |
| 0.529 s (direct) | — | 495 ms |
The arithmetic closes: 549 + 456 = 1005 ms against 1.041 s measured at the shell.
Three things fall out:
The overhead is not a constant. Across the day's samples the classifier call ranged from 476 ms to 1449 ms, and the first request after a restart cost 1449 ms on its own. It is a full inference call and inherits whatever the backend is doing. Any single number you quote for it is the number you happened to catch.
Token cost is fixed and larger than you'd guess. Each classification sent
281–293 input tokens — the rubric — and returned about 10. Routing hi, a
one-token prompt, costs ~281 tokens of classification before anything answers.
The serving call slows down too. 456–806 ms when routed, against 495 ms direct for the identical request. The classifier appears to leave the backend busy for the request queued behind it. You would never see this from wall clock alone.
Paired wall-clock timing tells you that it got slower, never where. Decompose
into the classifier call and the serving call before you quote a figure — the
first honest number here was almost double the one a single time run suggested.
The trap worth knowing about
If the classification call times out, returns the wrong shape, or comes back empty, the router falls back to the heuristic scorer — the thing whose failures you just paid half a second to avoid. Silently.
The official docs name the way out: set classifier_fallback: default_model and
a timeout routes to a model you chose deliberately, rather than to the scorer
that sends hard questions to a 3B model.
Also worth checking rather than assuming: the docs give timeout_ms a default of
2000, while the package in our container defaults to 3000. Read the version you
actually installed.
So which one
| Heuristic | LLM classifier | |
|---|---|---|
| Latency added | sub-millisecond | 476–1449 ms |
| Extra API calls | none | one per request |
| Extra input tokens | none | ~280 per request |
| Hard-but-short prompts | routed down | routed correctly |
| Easy prompts | routed correctly | routed up |
| Fails by | being confidently wrong | timing out, then being confidently wrong |
The heuristic is the right default for high-volume traffic that looks alike, and
you can improve it a lot with custom_technical_keywords for your own domain
vocabulary. The LLM classifier earns its cost when prompts are varied, when
getting the model wrong is expensive, and when half a second doesn't matter —
batch work, agents, anything already taking seconds.
What neither of them is, is free.
Troubleshooting
The response says smart-router instead of a model name
return_raw_model_name isn't set. Without it the router echoes the alias you
asked for and you can't tell what served the request.
Every prompt routes to the same tier
Check the signals field in the routing decision. If it's null, no dimension
matched at all and the prompt scored 0.00, which lands in SIMPLE. That's a sign
your traffic doesn't use the vocabulary the default keyword lists expect.
Routing decisions aren't in the spend log
They're on the routed request, under metadata.routing_decision, not on the
classifier call. Filter with map(select(.)) as above — half the rows are the
classifier calls themselves and carry no decision.
The first request after a restart is much slower
Cold start. The first classification here took 1449 ms against roughly 550 ms once warm. Discard the first sample when timing.
What's next
The gateway now decides which model runs, and you know what that decision costs in latency and tokens. It still forwards every prompt verbatim to whichever model wins — including the ones containing customer names, emails and API keys.
The next post puts a filter in that path.