Re:What I Talk About When I Talk About Query Optimizer (Part...
by chrisjc on Monday January 29, 2024 @12:34PM
(#39179241)
Somewhat similar, see https://substrait.io/
So for example using DuckDB with the Substrait extension, if you create a table
create table t(a int);
and then query it as in the article, you can see something similar to what is described in the article CALL get_substrait_json('select * from t');
{"relations":[{"root":{"input":{"project":{"input":{"read":{"baseSchema":{"names":["a"],"struct":{"types":[{"i32":{"nullability":"NULLABILITY_NULLABLE"}}],...
DuckDB extension doesn't seem to cover any DDL operations though.https://duckdb.org/docs/extensions/substrait
Some other related discussions and links that i've collected over the years
https://news.ycombinator.com/item?id=37415494
https://news.ycombinator.com/item?id=34233697
https://news.ycombinator.com/item?id=31981568
https://datastation.multiprocess.io/blog/2022-04-11-sql-pars...
Re:What I Talk About When I Talk About Query Optimizer (Part...
by Sesse__ on Monday January 29, 2024 @05:39PM
(#39183758)
After a quick look, I'm not sure if I would call this “industrial strength”. In particular, the join optimizer (typically the heart of a large-scale SQL optimizer) looks very rudimentary? And the statistics it uses have zero idea about correlation, no histograms beyond min/max…
Re:What I Talk About When I Talk About Query Optimizer (Part...
by menaerus on Tuesday January 30, 2024 @03:20AM
(#39187603)
I was wondering about the same claim. However, I believe that JOIN's are a common weakness among OLAP database engines, and DataFusion is built on top of a columnar storage format - Apache Arrow.
Re:What I Talk About When I Talk About Query Optimizer (Part...
by sanxiyn on Tuesday January 30, 2024 @03:36AM
(#39187720)
By being columnar, I guess you could say DataFusion has a good executor, but no, not a good optimizer.
Re:What I Talk About When I Talk About Query Optimizer (Part...
by menaerus on Tuesday January 30, 2024 @04:41AM
(#39188116)
Not that I was trying to make any of those claims but just trying to correlate the domain with what appears to be a common problem in it.
Re:What I Talk About When I Talk About Query Optimizer (Part...
by biggestdummy on Monday January 29, 2024 @06:21PM
(#39184147)
There's also this pretty detailed article on StarRocks' query optimizer. (StarRocks is open source - focused on OLAP)
https://medium.com/starrocks-engineering/starrocks-inside-sc...
(can also use it in your own projects)
It is quite similar to what is described in this post