intermediatePart 2
LiteLLM complexity routing: the right model for each request, and what it costs in latency
· 12 min read
+
+0/5
🎯 Skill path0/5 earned
Self-hosting an LLM gateway
- 1One endpoint, scoped keys
- 2Route work to the right model
- 3Mask PII at the gateway
- 4Clean traces, untouched answers
- 🏆Prove it holds under load
Part 1 ended on an uncomfortable number. The same three-word answer cost 3 tokens from a small model and 200 from a reasoning model — which spent all 200 thinking and returned nothing at all.
Every request your apps send picks a model, and mostly that choice is made once, hardcoded, and never revisited. This post puts the gateway in charge of it instead: classify the request, route it to a model sized for the work. Then it measures what that decision costs, because it is not free and most write-ups skip that part.