Zero failures. Half the latency budget. $0.32 a million tokens.

Running inference in production is not the same as running a model on a single GPU. Once traffic grows, you are routing every request, scaling GPU-backed replicas, isolating tenants, metering usage, and holding a latency SLA, all while trying to keep the bill sane. That orchestration work is a different job from the inference engine itself.

Tensormux lives at that orchestration layer, the tier between your app gateway and the inference engine. It does not replace vLLM or SGLang; it runs on top of them and handles what an engine leaves out: LLM-aware routing, GPU-aware autoscaling, a distributed KV cache, and real multi-tenancy (per-tenant limits, metering, isolation). You bring your engine and your GPUs, and we get them serving more traffic, more reliably, to more tenants.

We kept this run deliberately controlled and easy to reproduce: same model, same engine, fixed capacity, and only one variable changing between runs, the routing strategy. No custom runtime, no quantization tricks. Just the platform doing its job on a standard stack.

Pass criteriaA run passes when every routing strategy holds p95 TTFT under the 1000 ms SLA with zero failed requests. The four checks above are the result of this run against that bar.

Why routing is where a control plane earns its keep

Where a request lands is not a small detail. Modern engines hold a KV cache of the work they have already done, so when a request reaches a replica that already has its prompt prefix cached, the engine skips the expensive prefill step and the first token comes back quickly. Send that same request to a cold replica and it repeats all of that work. Multiply this across shared system prompts, RAG context, long chat histories, and many tenants on the same GPUs, and routing becomes one of the biggest levers you have over both latency and cost.

This is what Tensormux is built around. It ships prefix-cache-aware routing next to load-based strategies like least-request and least-latency, so each request goes to the replica most likely to answer it fastest.

This first benchmark measures the floor, not the ceiling. The workload here is uniform with no shared context to reuse, and all four replicas are identical, so there is genuinely nothing for a smart router to exploit. That is the point. Before we show routing pulling ahead on the workloads where it should, shared prefixes and multi-tenant contention, both next in this series, we wanted to confirm the boring but essential thing first: every routing mode stays stable and holds the SLA under sustained load.

That cost is the fully-loaded number for serving on your own H100 at this latency-optimized point, and it drops further at higher utilization, the usual trade between latency and cost. It is priced on GPU time, not resold through an API. The value is that it is a known, predictable number on hardware you control.

Before anything clever, the unglamorous thing has to be true: the platform holds a real SLA, stays reliable, and costs what you expect, on standard hardware and a standard engine. That is what this run shows, and it is the foundation everything else builds on.

This run validated the foundation. The next two go straight at the part a single-tenant throughput test cannot show, where the control plane actually pulls ahead:

• Prefix-aware routing on shared-prefix workloads (system prompts, RAG, multi-turn), where the routing decision cuts TTFT by a lot rather than a little.
• Multi-tenant SLA isolation, holding every tenant's SLA even while one tenant spikes, the noisy-neighbour test, and how many models you can pack onto a GPU while still meeting SLA.