MoE vs. dense models: What you need to know before picking a model
Saw a comment online about Models getting dumber or Model Labs trying to just pass benchmark as a tradeoff for losing some capabilities.
This brings back memory of model selection in early days where I mostly pick the current 'hottest' model for my workflow. Often the wrong place to start.
The process is still common and hasn't changed much for most product teams. The team pulls up a benchmark leaderboard. They compare MMLU scores, context windows, pricing per token. Someone argues for the frontier model, other argues for cost and eventually pick one and move on.
Two architectures. One decision most PMs make without knowing what exist.
MoE vs dense
A dense model activates all its parameters on every request. Every token that comes in gets processed by the full network i.e every layer and weight every time. GPT-4 in its original form, Llama models, most of the models PMs interact with daily are dense.
A Mixture of Experts model works differently. Instead of one large network that processes everything, it has multiple specialised sub-networks AKA the experts and a routing mechanism that decides which experts handle each token. On any given request, only a fraction of the total parameters activate. DeepSeek-V3, Mixtral, and increasingly the frontier models shipping recently are MoE.
The number that may confuse people is a MoE model might have 141 billion total parameters but only activate 22 billion on any given request. The total parameter count and the active parameter count are different numbers, and they mean different things for your product.
Why this matters for inference and therefore for product decisions.
Inference cost and latency are driven primarily by active parameters, not total parameters. A MoE model with 141B total but 22B active is cheaper to serve per request than a dense model with 70B parameters because it's doing less compute per token despite having a larger parameter footprint.
This might sound counterintuitive as bigger total parameter count may be lower per-request cost for MoE.
This is where the product opportunity lives but it comes with trade-offs.
Memory: A MoE model loads all its experts into memory even though it only uses some of them per request. A model with 141B total parameters needs to fit all 141B worth of weights somewhere, even if only 22B activate at inference time. The memory footprint is determined by total parameters. The compute footprint is determined by active parameters. For deployment, you need enough memory to hold the whole thing.
Latency profile: Dense models have a more predictable latency profile. The same computation happens on every request. MoE models introduce routing overhead and can behave differently across request types, some tokens route to experts that are slower, some to ones that are faster. For latency-sensitive features, understanding this variability matters.
Consistency: Dense models tend to be more consistent across similar inputs. MoE routing can send similar inputs to different experts depending on token-level decisions, which can produce more varied outputs. For use cases where consistency matters e.g structured extraction, classification, evaluation, this is worth testing explicitly.
The practical product read
If you're calling a frontier API and not running your own inference, MoE vs dense is mostly abstracted away. You're paying per token regardless of what's happening under the hood. The benchmark performance and the price are what you're buying.
Where it becomes a real product decision is in three situations.
Selecting open weights models for local or sovereign deployment: If you're choosing between Mixtral-8x7B and Llama-3-70B for a self-hosted deployment, you're making a MoE vs dense decision. The MoE model will need more memory but cost less per request at scale. The dense model will be more predictable. Which matters more depends on your hardware constraints and your traffic volume.
Evaluating cost at scale: If you're at the stage where per-token cost is a meaningful line item, understanding whether the model you're using is MoE or dense helps you model your infrastructure costs more accurately. Active parameters drive compute cost. Total parameters drive memory cost. Knowing which you're constrained by helps you pick the right model.
Latency-sensitive features: If your feature has tight TTFT requirements, a dense model's predictable latency profile is an advantage. MoE routing adds a small overhead and introduces variance. For a chat assistant where 400ms matters, that variance is worth knowing about before you commit to an architecture.
What to add to your model evaluation process.
Is this model dense or MoE?
What are the total and active parameter counts?
And given my deployment constraints; memory, latency, consistency requirements; which architecture actually fits what I'm building?
Asking the right product questions helps make better infrastructure decisions before the engineering team is already committed to the wrong model.