Why evals are the new A/B tests
For a long time, A/B test was one of the methods by which product teams answered questions around what good looks like. You shipped two versions, waited for traffic, and read the result. It worked wel
This is for people trying to run AI models locally on a personal machine. Say you want to run Llama locally on a MacBook Air, you visit Hugging Face and see a number printed on every model card that a
Saw a comment online about Models getting dumber or Model Labs trying to just pass benchmark as a tradeoff for losing some capabilities. This brings back memory of model selection in early days where
When building AI products, the fastest and laziest way is to pick a model the way you typically pick a cloud provider. Evaluate a few options, choose one, and move on. One model, One API and everythin
I remember when prompt design was one of the hottest topics in getting value out of LLM For teams building AI features, it sits in the engineering backlog, or the AI team's to-do list, or nowhere at a
Engineers building AI/ML should understand continuous batching intuitively. Some PMs have never heard of it or don't understand it. That gap costs product teams real money and slower user experiences,
There's a pricing detail most product teams don't look at closely enough. Input tokens and output tokens are not the same price. Output tokens cost more. Sometimes 2-3x more depending on the model. Th