News for hackers. Stuff that matters.
Re:Next-Level Benchmarking Overfitting – Thomas Neumann by lmeyerov on Wednesday November 11, 2026 @10:30AM (#50454594)
We have been building GFQL for vectorized CPU/GPU cypher (think duckdb for graphs) with agentic assistance, and our thinking is this is a stepping stone to something even more aggressive

Today, we are embracing doing hotpath specializations for patterns that would have been too expensive to maintain before. LLMs keep that cost bearable, and being able to do that for important users (like ourselves) is a superpower. Our benchmarks are beating the big guys - and importantly, on anything we care about.

Where we really want that to go is runtime / loadtime specialization. Our customers would absolutely want their top queries autotuned. DB research

Re:Next-Level Benchmarking Overfitting – Thomas Neumann by artrax on Wednesday November 11, 2026 @10:57AM (#50454911)
How would you find out which optimizations to apply to which queries? One decision that brings a 10x improvement, might slow down a very slightly different query by 100x.

Thats the core issue of database query optimization, and thats why (good) optimizer target the 99% percentile, not the peak, or the average.

Not blowing up is the most important criteria to be able to rely on a system.

Re:Next-Level Benchmarking Overfitting – Thomas Neumann by benjiro29 on Wednesday November 11, 2026 @11:38AM (#50455389)
Your argument applies to literally every database. A sequential scan can be way faster then a using a index. That is a optimization that can under the right circumstances result in a great benchmark score, but a bad experience for some customers but great for others.

The idea that benchmark optimizations are all bad, is the flawed argument. A benchmark is nothing else but a pattern. You can indeed over optimize like having a LLM recalculate values in advance but if those optimizations also work on non-benchmark SQL prompts, ... That brings us back to the original point.

There is nothing wrong with having LLMs try and find spot to optimize, as long as your test pattern is not limited to a singular set of data. The reality is, there is actually a lot of meat on the bones in a lot of databases, because it was not physically possible to try and run 100 or 1000s of permutations on query/data patterns. That has now changed.

A discussion like this kind of reminds me driver optimizations for GPUs. The idea of only targeting the average, is what GPU's did like 15+ years ago. Now nobody even blinks about this anymore, and people even demand game specific optimizations.

Where optimizations that made for faster benchmark results, also applied to games getting more FPS. This was such a hot topic years ago. Now nobody blinks a eye anymore when we even see game on game optimizations.

I can even argue the other way...

If a client has a database with a know pattern, and that client can enjoy a 10x performance increase because of that pattern optimization. And it does not reduce performance for others. Maybe that same optimization will trigger on some other client, who was not even the target.

What if your database has optimization that makes Wordpress work way faster. Thing is ... this type of targeted optimization already happens everywhere. I find it just odd that this is a discussion regarding databases.

Pre rendering data for a benchmark is indeed cheating, because that does not help clients, its misleading. But being able to show, "hey, we make X popular software faster then (example) postgres". The client are those that gain. And with LLMs that is now possible to do, without the cost the limited this in the past.

Re:Next-Level Benchmarking Overfitting – Thomas Neumann by artrax on Wednesday November 11, 2026 @12:58PM (#50456362)
Yes, I am aware that databases do that. But my point is that databases are conservative with what kind of optimizations they do, because the space of possible queries are infinite, and they want to have a guaranteed bound on what the worst case behavior is.

Big OLAP vendors like Snowflake do have the workloads of their customers, but you cannot just evaluate your optimizations on all workloads because its too expensive. You will only notice after deployment, when the customer calls you to complain. Or worse, they won't even consider you, because your novel workloads are too slow.

My point is not that specialized implementations can't deliver big performance gains, but that deciding when to use optimizations that have downsides reliably is very hard. Even for a classical problem, well isolated problem, like join-order optimization, even if you have perfect cardinality estimates (which you don't), its still NP-hard, and still, even optimizers that are considered advanced, will choke.

Anything engine/implementation specific is far worse, with smaller gains and much bigger downsides, which every systems routinely screws up because its just really difficult to decide when to use what.

Re:Next-Level Benchmarking Overfitting – Thomas Neumann by lmeyerov on Wednesday November 11, 2026 @02:06PM (#50457166)
a lot of times customers have key queries and query families, so while the problems are hard in theory, they're not in practice. the more general problem is the wrong one. likewise, teams are already using agents to profile and optimize workloads and do custom db extensions, so boundaries like human/machine and vendor/user are changing as-is.
Re:Next-Level Benchmarking Overfitting – Thomas Neumann by pixl97 on Wednesday November 11, 2026 @12:46PM (#50456200)
>might slow down a very slightly different query by 100x.

Again, any optimization needs testing and QA. But in general shouldn't things like pinning keep the plan to a single query type?

Also 99% of what?

If you have a query that runs once a day, but takes a very long time to run, a 10x improvement is huge.

Re:Next-Level Benchmarking Overfitting – Thomas Neumann by lmeyerov on Wednesday November 11, 2026 @02:01PM (#50457123)
you said the magic words, query planner

the point is the query planner can have more hotpaths available to it than historically possible, and richer optimizations along them than used to be feasible