Why your agent's memory is a billing problem

There is an interesting question I tackle most times when I am thinking through agent memory. Is it a capability question, a product question, a design question, or an experience question? The more I build in this space, the more I realise it is not one-size-fits-all. Agent memory is encompassing and cuts across multiple dimensions of the product development lifecycle.

Questions like, can it remember what happened last session? Can it recall a decision made three steps ago? Can it hold context across a long task? Those are the right questions to ask as a product builder.

The answers often carry real financial cost. So I started adding a sanity check in my initial design sprints: what does remembering actually cost, and who is paying for it?

Here is what happens under the hood.

When an agent remembers something, it is not storing a note in a database and retrieving it cleanly later. At inference time, memory means tokens. Every piece of context the agent needs to act, conversation history, prior decisions, tool outputs, retrieved documents, system instructions has to be present in the context window when the model generates the next step.

Every time, that context costs money and time to process on every single inference call.

A task that runs for ten steps is processing the accumulated memory of every prior step at each new step. By step ten, you are paying to process nine steps worth of history just to generate step ten. The cost may not be linear and often compounds quickly.

This can create a billing problem that may be too late to discover and other hidden costs impacting user experience

It gets more specific than that.

There are four ways agents handle memory, and each has a different cost profile.

In-context memory is the simplest. Everything the agent needs is in the context window. Fast to retrieve, zero lookup latency, perfect recall. Also the most expensive because every token costs money on every call, and as the task grows, the context grows with it. This is the default for most agentic products and one of the reasons infrastructure bills may surprise teams at scale.

External memory retrieval is the alternative. Store information in a vector database or knowledge store and retrieve relevant chunks at runtime. Cheaper per call because you are only loading what is relevant, not the full history. But retrieval is imperfect. Relevant chunks could get missed, it can add noise, and retrieval itself takes time and adds latency to every step.

Summarisation is a middle path. Compress older context into a summary, keep the summary in context instead of the full history. Cheaper than full history, but you lose precision. The summarisation step itself costs tokens. And the model is now reasoning on a lossy representation of what actually happened.

Structured memory is the most engineered approach: Extract key facts and decisions into structured records, retrieve them explicitly when needed. This is precise, cost-effective, and scalable. Also the most work to build and the most fragile when the extraction logic misses something.

None of these methods is right for every agent. Most production agents need a combination and the choice of which to use where is a product decision that directly affects your cost model and user experience.

Where this becomes a product failure mode.

Treating memory as a feature spec and not an embedded spec is a recipe for surprises down the line for agent infrastructure costs, experience and compliance. It is easy to build an agent that remembered everything because that felt like the right experience. However, you need to model what remembering everything costs at the concurrency levels you are targeting.

A support agent that holds full conversation history across a ten-turn resolution is not the same cost as a support agent that summarises after every three turns and retrieves structured records for relevant prior cases. The user experience might be nearly identical. The infrastructure cost can differ by an order of magnitude.

That trade-off is a product cal, If the relevant people are not making it, the default is always the expensive version and more trouble down stream

The agentic memory stack is also where context window decisions interact with serving decisions in ways that are easy to miss.

Longer context windows on frontier models made it technically possible to keep more in memory. That created a temptation to use them as a substitute for proper memory architecture. Why build retrieval when you can just load everything? Because at scale, the cost of loading everything into a 128k context window on every inference call is substantial. The KV cache for a 128k context grows accordingly. At high concurrency, KV cache pressure limits how many parallel requests your infrastructure can serve. Your memory design is now constraining your throughput ceiling.

It may appear to be separate problems to solve, but they are the same problem at different levels of abstraction.

The question to ask before designing any agent’s memory.

What does this agent need to remember, at what fidelity, for how long, and what is the cost of that memory at the inference frequency and concurrency we are targeting?

If you have not answered that before you build, you will answer it when there is a major problem down the line.

Agent memory is not a capability. It is an architecture and every architecture choice has a price ranging from cost to future product development constraints.