# Small model, big model: routing as a product strategy

When building AI products, the fastest and laziest way is to pick a model the way you typically pick a cloud provider. Evaluate a few options, choose one, and move on. One model, One API and everything routes through it.

That decision makes sense early because It's simpler and faster to ship. But as you scale, it starts to cost you; in money, in latency, and in cases where a smaller, faster model would have served the user better anyway.

Model routing is the practice of sending different requests to different models based on what each request actually needs. It's not a new idea. But most product teams treat it as an infrastructure optimisation when it should be part of product strategy.

The principle is simple, not every request needs a frontier model. A user asking a factual question with a clear answer doesn't need the same model as a user asking for a nuanced multi-step analysis. A support ticket classification doesn't need the same compute as a contract draft. But in most AI products, both requests go to the same model at the same price. Model routing closes that gap.

![](https://cdn.hashnode.com/uploads/covers/5fd4714295130967e09927bf/6abeafa1-ebc0-4e5e-bd19-740522082f4b.png align="center")

A small model handles the simple request as it is faster, cheaper, good enough. The frontier model handles the complex one for slower, more expensive, genuinely needed. The user experience improves on both ends. The fast path gets faster. The expensive path is reserved for where it actually matters.

### **Routing has three product dimensions worth understanding.**

**Cost:** Small models are significantly cheaper per token than frontier models. For high-volume features where most requests are routine e.g classification, summarisation, simple Q&A; routing a fraction of traffic to a smaller model can reduce infrastructure cost meaningfully without touching output quality for those cases. The savings compound at scale.

**Latency:** Smaller models generate tokens faster. For features where TTFT matters; conversational interfaces, inline writing assistance; routing simple requests to a smaller model is a UX decision as well as cost decision. The fast path feels more responsive to the users.

**Quality preservation:** Routing also protects quality on the complex end. When the frontier model handles everything, its capacity is diluted across simple requests it's overqualified for. When routing separates traffic by complexity, the frontier model is reserved for the cases that actually need it. You get better outcomes on hard requests, not worse ones.

### **The routing decision is a product decision before it's an engineering one.**

The questions that matter aren't which models to use, that's an engineering call. **The questions that matter are:**

*   what does this feature actually need from the model on each request?
    
*   What's the distribution of simple versus complex requests?
    
*   What's the acceptable quality floor for the simple path?
    
*   What does the latency profile need to look like for users to feel the difference?
    

If the PM isn't asking them, the routing strategy defaults to whatever the engineering team assumes, which is usually "route everything to the best model we have" because that's the safe choice. Safe isn't the same as right in most cases

### **Three patterns worth indexing for.**

**Complexity routing:** Classify the incoming request before sending it to a model; is this simple or complex? A lightweight classifier (sometimes another small model, sometimes a rule-based system) makes the call and routes accordingly. The classifier cost is minimal compared to the savings on the main model.

**Fallback routing:** Start with the small model. If the output doesn't meet a quality threshold; measured by confidence score, format validation, or a secondary evaluation, escalate to the frontier model. The user gets the fast path when it works and the quality path when it doesn't.

**Task routing.** Different features on the same product need different models. A search feature and a document drafting feature have completely different requirements. Routing at the feature level, not the request level, is the simplest version of this strategy and often the most impactful starting point.

![](https://cdn.hashnode.com/uploads/covers/5fd4714295130967e09927bf/0bb33a75-603e-4d47-8771-0fee8907858a.png align="center")

### **During your next product review, remember to ask**

For every AI feature we ship, are we sending every request to the right model or just the most expensive one we have access to?

If the answer is the latter, routing is the lever you haven't pulled yet.