It's not like there's a shortage of skills in this area, it seems like our one specific industry just has a weird blindspot.
When I talk about this with other CS people in my own country (Spain) they tend to refer similar experiences.
“The customers want lower latency at 30% load for unique queries.”
“Err… we can scale up for more throughput!”
ಠ_ಠ
Whatever happened with Llama 4?
This isn't just that nobody cares about the truth. People 100% care! If you actually degrade a performance metric as measured post-hoc in full prod, someone will 100% notice, and if you want to keep your feature un-rolled-back, you are probably gonna have to have a meeting with someone that has thousands of reports, and persuade them it's worth it to the business.
But you're always gonna have more luck if you can have that meeting _before_ you degrade it. But... it's usually pretty hard to figure out what the exact degradation is gonna be, because of the things in my previous comment...
Sort of tangential, but as someone currently taking an intro statistics course and wondering why it's all not really clicking given how easy the material is, this for some reason makes me feel a lot better.
Pair that with skipping all the important problems (what is randomness, how do you formulate the right questions, how do you set up an experiment capable of collecting data which can actually answer those questions, etc), and it's a recipe for disaster.
It's just an exercise in box-ticking, and some students get lucky with an exceptional teacher, and others are independently able to develop the right instincts when they enter the class with the right background, but it's a disservice to almost everyone else.
Ultimately we are measuring extremely measurable things that have an objective ground truth. And yet:
- we completely fail at statistics (the MAJORITY of analysis is literally just "here's the delta in the mean of these two samples". If I ever do see people gesturing at actual proper analysis, if prompted they'll always admit "yeah, well, we do come up with a p-value or a confidence interval, but we're pretty sure the way we calculate it is bullshit")
- the benchmarks are almost never predictive of the performance of real world workloads anyway
- we can obviously always just experiment in prod but then the noise levels are so high that you can entirely miss million-dollar losses. And by the time you get prod data you've already invested at best several engineer-weeks of effort.
AND this is a field where the economic incentives for accurate predictions are enormous.
In AI, you are measuring weird and fuzzy stuff, and you kinda have an incentive to just measure some noise that looks good for your stock price anyway. AND then there's contamination.
Looking at it this way, it would be very surprising if the world of LLM benchmarks was anything but a complete and utter shitshow!