A router that can't see the conversation can't classify "yes"
Classifying the word "yes"
A model router's job is to read a request and decide which model should answer it. Cheap questions go to a small model, hard ones to a large one, and the bill comes down. The whole arrangement rests on being able to tell the difference.
Then a user types "yes".
Or "continue". Or "do it". Nothing in those two or three characters says whether the work being approved is a spelling fix or a database migration. A router scoring the current message in isolation sees a very short string with no technical vocabulary, and does the obvious thing: cheapest model.
Which means if you route requests to save money, your cheapest tier is probably absorbing work it should never have seen — and your savings figure is partly fake. On 4 August LiteLLM published a benchmark that measures both halves of that: how wrong the routing gets, and what it costs to fix.
The question generalises past their implementation — anything classifying a turn in a conversation has this problem — and the reason it's worth reading is that they measured the price of the fix, not only the benefit.
Everything below is LiteLLM's own measurement, published on their blog and quoted
here: v1.97, classifier gpt-5.4-mini, their three datasets, their reference
labels. None of it was run on our infrastructure, and we have not reproduced it.
The measurement
The sweep: 5,600 live classifier calls against real providers, described as
"two sweeps of seven configurations each (classifier_context_window_size of 0,
1, 2, 3, 5, 8, 10), one with assistant turns in the window and one without, across
three multi-turn datasets, with two repeats per conversation."
The variable is how many prior turns the classifier gets to see. Zero means it judges the current message alone. Ten means it reads the last ten turns first.
Agreement with reference tiers, by window size:
| Prior turns | Short-reply follow-ups | MT-Bench 2nd turns | ShareGPT multi-turn |
|---|---|---|---|
| 0 | 50.0% | 49.4% | 83.8% |
| 1 | 71.2% | 53.1% | 84.4% |
| 2 | 87.5% | 53.1% | 90.6% |
| 3 (default) | 85.0% | 55.0% | 91.9% |
| 5 | 86.2% | 53.8% | 91.9% |
| 8 | 87.5% | 55.6% | 91.2% |
| 10 | 90.0% | 55.6% | 88.8% |
Figure 1. Agreement with reference tiers by context window, drawn from the figures published in LiteLLM's benchmark. The red line is the 36-follow-up subset whose difficulty only resolves against history.
Three different stories in three columns, which is the first useful thing here.
ShareGPT starts at 83.8% and gains eight points. MT-Bench barely moves at all — 49.4% to 55.6% across the whole sweep — and their own caveat explains why: "MT-Bench's ceiling reflects its reference labels rather than router behaviour." The short-reply set is where it bites, 50% to 90%.
The number worth quoting, with its denominator
Inside that first column sits a subset, and it is the sharpest result in the post:
Agreement there is 14% at N=0, 47% at N=1, and 78% at N=2, and flat from there out to N=10.
Fourteen per cent to seventy-eight per cent, achieved by showing the classifier two prior turns.
That figure describes 36 follow-ups — the ones "whose final turn only resolves against the history". It is a deliberately selected hard subset, not a general accuracy claim, and reading it as "routers are 14% accurate" would be wrong. What it does show is the shape of the failure: when a turn's difficulty lives entirely in what came before it, a context-free classifier is worse than guessing, and almost all of the recovery happens by the second turn of history.
The curve going flat at N=2 is the practically useful part. This is not a "more context is better" result. It is a "two turns is nearly all of it" result.
What it cost, and what it cost more
This is the half that makes the finding actionable, and where the honest answer is more interesting than "it's cheap".
The window itself is free. Every paired comparison against the zero-window configuration has a 95% bootstrap confidence interval straddling zero. Window 1 with assistant turns off comes in at −17.8 ms, interval −68.9 to +29.6. Their explanation is that the window adds prefill only — the output stays a small fixed tier label. Over a 318-to-1,043 prompt-token range, tokens and latency correlate at r = 0.007.
The classifier is not free. Adding history costs nothing, but asking a model
to classify at all costs "p50 sits near 600 ms in every configuration" — and
that sits in front of the real completion. The classifier here is
gpt-5.4-mini, and it runs at $0.31–$0.61 per 1,000 requests across the whole
sweep. Cheap in money, half a second in time.
And accuracy raised the bill. This is the part worth sitting with. For the short-reply set, the modelled routed cost rose with the window — from $2.87 to $6.49 per 1,000 requests. Not a regression: the reason is "a tier mix of 66% SIMPLE at N=0 against 30% at N=2". Two thirds of those follow-ups were being served by the cheapest model. Once the classifier could see what they were approving, they stopped being.
So the routing was cheap because it was wrong. The saving was a bill someone else was paying, in answer quality, on requests nobody was auditing. On the other two datasets the effect runs the other way — ShareGPT's modelled cost falls slightly, because context lets the classifier settle ambiguous prompts into MEDIUM instead of defaulting up to REASONING.
That is the useful shape of it: better context doesn't reliably cut spend. It moves spend toward where the work actually was.
Two things this doesn't settle
The default is 3, and 3 is not the best number in any column. Ten is best for short-reply follow-ups at 90.0%, and simultaneously the worst result for ShareGPT past N=2, dropping to 88.8% from 91.9%. More history is not monotonically better, and the shipped default is a judgement call across datasets that disagree.
And they name their own limits, which is the mark of a benchmark worth trusting: "Reference tiers are judgement calls", "Routed completion cost is modelled rather than billed", "One classifier model was swept", and "Latency was measured on a single VM at concurrency 10". Concurrency 10 on one machine is not production, and a heavier classifier model carries more absolute latency than the one they used.
Why it matters if you route anything
The lesson survives the specific implementation. If you are choosing models per request — with a heuristic, a classifier, or a hand-rolled rule — the turns that will embarrass you are not the long technical ones. They are the short ones. A router scoring "yes" as a trivial request routes the approval of an architecture migration to the smallest model you own, and nothing in your logs will flag it, because a cheap answer to a cheap-looking question is exactly what you asked for.
Their fix is a window of prior turns. The cheaper fix, if your classifier has no such option, is to notice that short replies in an established conversation are the population to worry about, and to stop scoring them on their own contents.
We took the same router apart from the other direction recently — the seven scoring dimensions, the arithmetic on a real prompt, and what a free keyword scorer misses that a model catches — in route by complexity.