Why evals are the new A/B tests
For a long time, A/B test was one of the methods by which product teams answered questions around what good looks like. You shipped two versions, waited for traffic, and read the result. It worked well because the thing you were testing usually held still.
AI features don't hold still and that's why the question now gets answered earlier, by something called an eval.
When I quantized Qwen3-0.6B to 4-bit, perplexity went from 32.79 to 35.45. An 8.1% increase. I knew that before a single user touched the model. I don’t need a requirement for traffic split or waiting for significance. A fixed test set, a score, and an answer I could act on in the same session.
That's when it clicked for me. Evals now sit where A/B tests used to sit in a product team's workflow.
What A/B tests were good at
An A/B test answers one question well. Given two versions of something, which one do real users respond to better?
It works because the thing under test is usually stable. A button is green or blue, and the same user sees the same button every time.
Most AI features break most of those assumptions. The output changes from one request to the next, even with the same input. The thing under test also changes constantly. A prompt edit, a model swap, a new quantization level and a routing rule can each move quality, and you'll make several of those changes in a month. If everyone needed a traffic-based experiment, you would likely spend the whole roadmap waiting.
Where A/B tests fall short for AI
A feature that is wrong in a small share of cases still looks fine on average. A support agent that gives the wrong refund policy to one customer in fifty won't move your headline metric. But that customer remembers. Averages hide exactly the failures you care most about, and by the time they show up in your numbers, it may already be too late.
Some decisions can't be tested on users at all. When I fine-tuned a small model to propose decisions inside Boxy, I couldn't put a half-trained version in front of people and see what happened. I needed to know it was good enough to show first.
For instance, 3 scenarios
A prompt edit: You tighten the system prompt to cut token cost. An A/B test needs weeks of traffic to detect a small quality drop. An eval runs the new prompt against your saved cases in minutes and shows you which ones got worse.
A model swap:** You move from a larger model to a smaller one to save money. This is the decision with the most downside, because quality can slip in ways that are difficult to flag with a dashboard. An eval shows exactly which kinds of requests the smaller model fails, before any customer interact with it.
A quantization or routing change: The change is invisible in the interface. Users can't tell you what's different, because nothing looks different. Only a score tells you the model got 8% less certain. This is where an A/B test will struggle
In all three examples, the eval answers the question first and the A/B test only becomes useful once the change has already cleared a bar.
What an eval actually is
An eval is a set of inputs with a way of scoring the outputs. You run it before shipping to production, and again on every change.
The inputs come from real failures, edge cases you're afraid of, and examples that define what good looks like for your product. The scoring can be an exact match, a rubric, another model acting as judge, or a human reading a sample. Tools like Promptfoo make it easy to run on every prompt change.
It's closer to a regression test than an experiment, with one important difference. A regression test tells you whether something broke. An eval tells you whether quality moved, in which direction, across the cases you chose to care about.
I don't think evals replace A/B tests. They answer different questions at different points in the cycle.
An eval tells you whether a change is good enough to ship. An A/B test tells you whether users value it once it's out. Skip the eval and you test on your customers. Skip the A/B test and you assume your eval set matches what people actually do.
The order is what has changed. For a traditional feature, the experiment came first because it was the only way to learn anything. For an AI feature, the eval comes first because it is cheaper, faster, and catches the failures that experiments average away.
The eval set is the spec
Writing an eval set forces you to decide what a good answer looks like, which cases matter most, and what you won't accept. That is product work. Leave it to engineering and you get an eval set that measures whatever was easy to measure, which means your product may ends up defined by someone else's guess.
A practical place to start is twenty real examples where the feature failed or nearly did. Write down what the right output should have been, and run every change against them. The set is small and will be wrong in places, but it will already tell you more than most teams know about their own feature.
Before the next prompt change goes out, ask what you would need to see to call it better. If you can't answer that, you don't have an eval yet.
About the author
- I lead the strategy and delivery of products and systems that solve real problems, support commercial goals and scale across teams and markets. My work spans platform foundations, user-facing experiences, commercial tools and developer products, and I adapt my approach to the needs of each domain. My experience covers early-stage, scale-up and enterprise product environments, combining hands-on delivery with team leadership and award-winning work recognised across several innovative organisations.
Comments
Join the discussionNo comments yet. Be the first to comment.

