# Context windows are not free : What your PM never told your eng team

At first glance, you think there is no price tag for Nobody context windows.

It's not listed in the model pricing. It doesn't show up in your planning. Your engineering team probably hasn't flagged it as a constraint. But every token you put into a context window costs something; in compute, in latency, and in money, and most product teams are spending it without thinking.

This is the conversation that doesn't happen often enough among product teams

* * *

### Start with what a context window actually is.

When a model generates a response, it doesn't just see your latest message. It sees everything — the system prompt, the conversation history, retrieved documents, tool outputs, examples you fed it. All of that is the context window. The model reads all of it before generating a single token.

That reading is prefill. And prefill isn't free.

Every token in that window consumes compute proportional to the length of the input. Longer context means more compute, more memory, higher cost per request, and slower TTFT. The relationship isn't linear either, attention mechanisms scale quadratically with sequence length. Double the context, more than double the cost.

* * *

### Here's how it plays out in practice.

A product team decides to build a document assistant. Users upload reports, ask questions, get answers. Sounds simple. The PM writes the spec. Engineering starts building. Nobody asks how much of the document ends up in the context window on every request.

The answer, in a naive implementation, is all of it.

Every question the user asks sends the entire document back through prefill. A 50-page report might be 25,000 tokens. At current API pricing, that's not trivial per request. Multiply it by your daily active users and the feature that seemed straightforward in planning is now a significant line item that nobody budgeted for.

![](https://cdn.hashnode.com/uploads/covers/5fd4714295130967e09927bf/b50d74c0-0828-4743-8c6b-1e64a06e3ec9.png align="center")

This is not an engineering failure. It's a product failure that compound the cost and experience to serve your AI features

* * *

### The context window has four cost dimensions worth understanding.

**Money:** Input tokens are priced lower than output tokens but they're not free. Long contexts on high-capability models add up fast at scale. This is a unit economics conversation that belongs in product planning, not just in the infra post-mortem.

**Latency:** Every token in the context window adds to prefill time. More context means slower TTFT. For a conversational feature, that matters directly to the user experience. The spec that says "summarise this document and answer questions about it" has an implicit latency cost the product team probably didn't model.

**Memory:** KV cache grows with context length. More context means more GPU memory consumed per request. At high concurrency, many users simultaneously, this limits how many requests you can serve in parallel. Your throughput ceiling is partly a context window decision.

**Quality:** Longer contexts can degrade model performance. Relevant information gets diluted by irrelevant tokens. There's a reason retrieval-augmented generation exists, stuffing an entire knowledge base into a context window isn't just expensive, it often produces worse answers than retrieving the relevant chunks first.

![](https://cdn.hashnode.com/uploads/covers/5fd4714295130967e09927bf/8492e71c-6dc6-4ade-a133-5c53a67188b4.png align="center")

* * *

### The decisions that manage this are product decisions.

**Chunking and retrieval:** Instead of sending an entire document, retrieve only the relevant sections per query. This is a product architecture decision that affects cost, latency, and quality simultaneously.

**Conversation pruning:** In multi-turn chat, do you keep the entire history or summarise older turns? That's a product trade-off between continuity and cost. Most teams default to keeping everything and discover the problem later.

**System prompt length:** System prompts run on every single request. A 2,000 token system prompt multiplied across millions of daily requests is a significant cost centre. Trim it like you'd trim any other resource.

**Context hierarchy:** What goes in the system prompt vs the user turn vs retrieved context? The order and placement of information in the window affects both cost and how well the model uses it. This is prompt architecture, and it's a product skill.

* * *

The question to ask before any AI feature ships.

![](https://cdn.hashnode.com/uploads/covers/5fd4714295130967e09927bf/3b893318-482b-4cf9-b668-6ba287e29663.png align="center")

What goes in the context window on every request, how long is it, and what does that cost at the scale we're targeting?

If you can't answer that, you're guessing. And you'll find out the hard way, in an infrastructure bill that surprised nobody except the people who should have seen it coming.

Context windows are not free. They're a resource. Treat them like one.