Re:Next-Level Benchmarking Overfitting – Thomas Neumann
by lmeyerov on Wednesday November 11, 2026 @02:06PM
(#50457166)
a lot of times customers have key queries and query families, so while the problems are hard in theory, they're not in practice. the more general problem is the wrong one. likewise, teams are already using agents to profile and optimize workloads and do custom db extensions, so boundaries like human/machine and vendor/user are changing as-is.
Big OLAP vendors like Snowflake do have the workloads of their customers, but you cannot just evaluate your optimizations on all workloads because its too expensive. You will only notice after deployment, when the customer calls you to complain. Or worse, they won't even consider you, because your novel workloads are too slow.
My point is not that specialized implementations can't deliver big performance gains, but that deciding when to use optimizations that have downsides reliably is very hard. Even for a classical problem, well isolated problem, like join-order optimization, even if you have perfect cardinality estimates (which you don't), its still NP-hard, and still, even optimizers that are considered advanced, will choke.
Anything engine/implementation specific is far worse, with smaller gains and much bigger downsides, which every systems routinely screws up because its just really difficult to decide when to use what.