News for hackers. Stuff that matters.
Re:What I Talk About When I Talk About Query Optimizer (Part... by Sesse__ on Monday January 29, 2024 @05:39PM (#39183758)
After a quick look, I'm not sure if I would call this “industrial strength”. In particular, the join optimizer (typically the heart of a large-scale SQL optimizer) looks very rudimentary? And the statistics it uses have zero idea about correlation, no histograms beyond min/max…
Re:What I Talk About When I Talk About Query Optimizer (Part... by menaerus on Tuesday January 30, 2024 @03:20AM (#39187603)
I was wondering about the same claim. However, I believe that JOIN's are a common weakness among OLAP database engines, and DataFusion is built on top of a columnar storage format - Apache Arrow.
Re:What I Talk About When I Talk About Query Optimizer (Part... by sanxiyn on Tuesday January 30, 2024 @03:36AM (#39187720)
By being columnar, I guess you could say DataFusion has a good executor, but no, not a good optimizer.
Re:What I Talk About When I Talk About Query Optimizer (Part... by menaerus on Tuesday January 30, 2024 @04:41AM (#39188116)
Not that I was trying to make any of those claims but just trying to correlate the domain with what appears to be a common problem in it.