Engineers building AI/ML should understand continuous batching intuitively. Some PMs have never heard of it or don't understand it. That gap costs product teams real money and slower user experiences, whether you're running your own inference stack or calling an API.
Here's why it matters and what you can actually do about it.
The problem with static batching
Before continuous batching existed, LLM serving worked like a London red bus with a fixed departure time.
You set a batch size, say, eight requests. The server waited until it had eight requests, processed them together, returned the results, then waited for the next eight. If only three requests came in, the server still waited. If one request in the batch needed 500 tokens and another needed 10, every request in that batch waited for the longest one to finish before anything returned.
This is static batching. It's simple and also wasteful.
The problem is that LLM requests are not uniform. A user asking for a one-line summary and a user asking for a full document draft land in the same batch and run together. The short request finishes in seconds. The long one takes minutes. Every other request in the batch waits regardless.
GPU time is expensive. Idle GPU time is money leaving the building and this affects user experience.
Continuous batching changes the contract.
Instead of waiting for a fixed batch to complete, the server fills GPU capacity continuously. When one request finishes generating, even mid-batch, a new request slots in immediately to take its place. The GPU never sits idle waiting for stragglers.
Think of it less like a London red bus with a fixed departure and more like a conveyor belt. Requests join when they're ready. They leave when they're done. New ones take their slot immediately.
The result is higher throughput at the same hardware cost. vLLM's original benchmarks showed 2-4x throughput improvement over static batching on the same GPU. This gain is the difference between one server handling 100 users or 400.
You don't run your own inference stack. This still applies to you.
If you're calling OpenAI, Anthropic, or Google's API, you might think continuous batching is someone else's problem. Not really, you're just on the other side of it.
The provider's infrastructure is running continuous batching across thousands of customers simultaneously. Your requests join that batch. Your output length occupies a slot in it. When you hit a rate limit or see TTFT spike during peak hours, you're experiencing batch saturation you can't see and can't control.
What you can control is your side of the equation. Shorter outputs, tighter prompts, smarter request patterns, these don't just reduce your token bill. They make your requests lighter to serve, which means they clear the batch faster, which means your users see better latency under load.
You don't need to run vLLM to benefit from understanding how it works.
Here's where it becomes a product problem.
Continuous batching is enabled by default in vLLM and most modern serving frameworks. Most product teams ship AI features without knowing it's there, which means they also don't know when they're working against it.
Two things product teams do that undermine continuous batching:
Generating unnecessarily long outputs. Every extra token a request generates is time it spends in the batch. A verbose system prompt that produces a 2,000-token response instead of a 400-token one is occupying a batch slot for five times as long. Other requests queue behind it. TTFT for those requests climbs. The user experience degrades for everyone on the server.
Poor concurrency planning. Continuous batching is most effective under load. A feature that generates requests in tight bursts, e.g a "generate all" button that fires 20 requests simultaneously will saturate the batch differently than a feature with steady incremental traffic. Thinking about your request pattern helps you understand how your feature behaves under real usage.
What this means for how you spec features.
Output length is a concurrency variable, not just a cost variable. A feature that generates long outputs doesn't just cost more per request, it occupies batch slots longer, which affects every other concurrent user on the same server. That's a shared infrastructure concern. Your feature's verbosity affects someone else's TTFT.
Request patterns matter. Know whether your feature generates requests in bursts or steady state. Burst patterns need different infrastructure planning than steady state. This is a product input, not just an engineering one.
Streaming is more than UX. Streaming output back to users as it generates is good UX, users see results immediately. It's also good for batch efficiency because long-running requests release their slot incrementally rather than holding it until full completion. Design for streaming on any feature with variable output length.
The question to ask before any AI feature ships
What is the expected request pattern, what is the expected output length, and how does that behave under the concurrency levels we're targeting?
If you can answer that, you're thinking about your feature the way the infrastructure thinks about it. Whether that infrastructure is a GPU cluster you control or an API you're paying per token. That's where the good product decisions live.
