Prefill vs. decode: Why the same model feels fast in one product and slow in another
There's a question that comes up a lot when teams start measuring AI performance.
"Why does our model feel slow? We're using the same model as everyone else."
The answer almost always lives in the same place. Not the model. Not the hardware. The phase of inference the feature spends most of its time in.
Generating a response from an LLM is actually two different jobs happening in sequence. They look connected. They're not. They have different cost profiles, different latency characteristics, and different implications depending on what your product is asking the model to do.
Those two jobs are prefill and decode.
Prefill is the model reading.
When a user submits a request; a prompt, a document, a conversation history; the model processes all of that input at once before generating a single output token. That processing is prefill. It's computationally intensive but it's parallel. The model reads everything in one pass, builds an internal representation of what it's been given, and populates the KV cache with everything it needs to generate a response.
Prefill is fast relative to the amount of work being done. A model can process thousands of input tokens in a fraction of a second on good hardware because the computation is parallelised across the entire input at once.
But there's a catch. The longer the input, the more expensive prefill gets. Context windows that are nearly full, long system prompts, retrieved documents stuffed into the prompt, all of that adds to prefill cost. And prefill cost directly determines TTFT. The user is waiting for prefill to finish before they see the first word.
Decode is the model writing.
Once prefill is done, the model generates output one token at a time. Each token depends on the previous one. There's no parallelism here. The model generates token one, feeds it back in, generates token two, feeds that back in, and so on until it hits a stop condition.
Decode is slower per token than prefill is per token. It's also where most of the wall-clock time goes for any feature producing long outputs. A model summarising a 10-page document is spending the vast majority of its time in decode, not prefill.
Tokens per second, the throughput metric, is a decode metric. When someone tells you a model generates at 80 tokens per second, they're talking about decode speed.
Here's where it becomes a product problem.
Different features are dominated by different phases. And most product teams don't know which one they're in.
A chat interface is prefill-sensitive. Every message the user sends extends the conversation history. As the conversation gets longer, prefill gets more expensive, KV cache grows, and TTFT climbs. The user doesn't know why the assistant feels faster at the start of a conversation than at the end of a long one. They just know it feels slower.
A document summarisation feature is decode-sensitive. The input gets processed once in prefill. Then the model writes for a long time. The user isn't waiting for the first word, they've already accepted they're waiting. What they care about is total time to completion. Optimising TTFT here is the wrong lever entirely.
A code completion feature is both. Short input, short output, low tolerance for any latency at all. The user is mid-keystroke. They're not waiting for a response. They're expecting the tool to keep up with them.
Same model with three features and three completely different bottlenecks.
The decision framework is straightforward once you see it.
Ask two questions about your feature. How long is the typical input? How long is the typical output?
Long input, short output: your feature is prefill-heavy. The thing to optimise is how fast the model processes context. KV cache reuse, prompt compression, reducing unnecessary context are your levers.
Short input, long output: your feature is decode-heavy. Throughput is the metric. Batching, speculative decoding, model quantization are your levers.
Balanced, you need to care about both, which usually means you need to be more deliberate about where you invest infra effort.
This is a product call before it's an engineering call. The infra team can tell you what's achievable. You need to tell them what the experience requires. Those two conversations don't happen often enough.
The reason the same model feels fast in one product and slow in another isn't a mystery. It's physics.
Prefill is parallel. Decode is sequential. Your feature sits somewhere on that spectrum. Where it sits determines what good performance looks like, what you measure, and what you ask your infra team to optimise.
If you don't know where your feature sits, you're optimising blind.