I remember when prompt design was one of the hottest topics in getting value out of LLM
For teams building AI features, it sits in the engineering backlog, or the AI team's to-do list, or nowhere at all. It's treated as something you figure out once and move on from. And it's likely costing product teams money they can't see.
Every prompt your product sends is a transaction.
Input tokens cost money and system prompts run on every single request. A 5,000-token system prompt sent to 10,000 users a day is 50 million tokens before a single user message is processed. At current API pricing across major providers, it becomes noticeable at scale.
Most product teams have never modelled it. They wrote the system prompt during development. It worked. They shipped it and didn't look at it again.
Prompt design has three cost dimensions that matter for product decisions.
Length: Every token in your prompt costs money on every request. Not just the system prompt, the few-shot examples you included to improve output quality, the formatting instructions, the contextual preamble that made sense during development. Each one compounds at scale.
Structure: How you frame a task to the model affects how many output tokens it generates in response. A prompt that invites verbose reasoning generates more tokens than a prompt that asks for a direct answer. Both might produce acceptable quality. Only one costs half as much.
Stability: Prompts that are consistent across users unlock prefix caching, a serving-level optimisation where the KV cache for identical prefixes is reused rather than recomputed. A system prompt that varies per user, per session, or per request breaks prefix caching and removes a significant efficiency gain.
The practices most teams can benefit from.
Token auditing: Count the tokens in your system prompt. Count the tokens in your average output. Model the cost per request at 1,000, 10,000, and 100,000 daily active users. Do this before you ship, not after you're surprised by a hefty bill.
Prompt versioning: Treat your system prompt like code. Version it. Review changes. Understand what a 200-token addition costs at scale before it goes to production. Right now most teams change prompts the way they change a Slack message, casually, without consequences.
Output length constraints: Set max_tokens deliberately. Know what the right output length is for your feature and enforce it. A prompt that produces 800 tokens when 200 would serve the user equally well is leaving cost on the table on every single request as well as degrading user experience with verbose
Output format control: Asking the model to respond in structured JSON when you need structured data is crucial, it bounds output length and removes verbose preamble. Formatting instructions in a prompt can serve as cost instructions.
The question to add to your feature spec.
What is the token cost per request for this feature, what does that look like at our target scale, and how does our prompt design affect both sides of that equation?
If your engineering team can't answer that when you ask, nobody has modelled it. That's a good conversation starter
Prompt design is not a developer experience problem. It's becoming a unit economics problem. The PM who treats it that way will ship AI features that stay profitable as they scale.
The PM who doesn't may find out the hard way, usually after it's already expensive to fix.
