Inferred 06: A new routing engine
A new routing engine
This week - the router. On Wednesday we shipped the biggest change to provider routing since OpenRouter began. Instead of mostly sending traffic to the cheapest provider, the router now weighs speed and reliability alongside price for every request.
Faster first touch. First tokens arrived sooner on 15 of the 20 models we tested, by as much as 42% on DeepSeek V3.1.
Fewer rate limits. The new router hit fewer 429s than the old one in 27 of 28 head-to-head blocks.
Faster streams. Output tokens per second rose on 14 of the 20 models, by as much as 88% on Qwen3.8 27B.
Across the whole experiment, median time to first token fell 16.9% and the share of requests hitting a 429 roughly halved, for 6.0% more cost per request.
Speedy initial tokens
Time to first token is of paramount importance not just to the experience of conversing with models but also in the long trains of multi-turn agent workflows. The new router improves that snappy touchpoint across most models. We score every provider for every model separately, so the gains can vary a lot by model. DeepSeek V3.1 got its first token 42% sooner. Qwen3 32B, Qwen3.8 27B, Kimi K2.6 and MiniMax M3 were each 20 to 28% faster.
Five models were flat or slower. Interestingly, these tended to be models where the various providers were already quite close. There was less room to improve across available inference and a provider that is cheaper but slower still saw a solid share of tokens.
Consistently reliable
To measure the new router’s reliability fairly, we switched back and forth between the two routers every three hours for a full week. The blue line is the new router and the gray line is the old price-first one. Each point is a 3-hour block.
The blue line sits below the gray one in 27 of 28 pairs, across weekdays, the weekend and overnight. The typical old-router block had about 6% of requests hit a 429 before succeeding. The typical new-router block had about 2.5%.
This improved reliability is downstream of an increasingly watchful eye on capacity. When we see a provider start to turn away requests, that provider receives proportionally less traffic until it recovers, no matter where it sat in the original routing order.
Faster streams
A quick first token is great, but what about the rest of the response? We also measured how quickly each model streamed once it started. This new router saw output tokens per second rise on 14 of the 20 models. Qwen3.8 27B streamed 88% faster, Llama 3.1 70B 81% faster and DeepSeek V4 Pro, the largest model in the experiment, 17% faster.
A few models moved the other way. DeepSeek V3.1 streamed 5% slower, though it had the biggest TTFT gain of any model.
These improvements do sometimes come at a cost. Across the experiment the new router cost 6.0% more per request on average. For traffic where cost matters most, you can add :floor to the model ID or set provider.sort: "price" on the request. You can also make price your default sort in account settings.
Explicit instructions vs letting it ride
A quarter of open-weight requests on OpenRouter still name their providers with provider.order, only or ignore, and another 11% set an explicit sort. Those requests will obviously not benefit from the new router improvements.
Plenty of those pins have good reasons behind them, such as data policies, contracts, or eval parity. But we expect a lot of them were originally set to avoid a slow or flaky provider. That’s now the router’s job! If you pinned for speed or reliability, try dropping the pin on one workload and compare. And on the off chance pinning was in order to maintain cache, don’t worry - once a conversation lands on a provider, it stays there while the cache is warm.
We’re hopeful that this new router provides a foundation for an ever-improving inference market. Providers can now optimize across multiple dimensions of quality rather than being artificially focused on cost alone (or even driven to playing pricing games with list vs effective price). We wanted to incentivize a market where competing providers won traffic by serving better inference, not just cheaper inference.
Onwards,
Peter
Charts 1 to 3 cover a switchback experiment from October 9 to 16, 2026 (UTC): 20 open-weight models, with the old and new routers taking turns in 3-hour blocks, about 24M requests split evenly between them. TTFT is the serving provider’s median time to first token on streamed requests. 429 and retry rates are the share of successful requests that hit a 429 or needed more than one attempt along the way. Throughput is median output tokens per second after the first token, on streamed requests.
Past issues of Inferred are on the web at openrouter.ai/data/newsletter.