There's a fair amount of overlap, but where the databricks article explains the techniques with charts and high level explanations, I go over the code instead.
Re:Cost-Based Optimizer in Apache Spark 2.2
by makmanalp on Thursday August 31, 2017 @11:54AM
(#15141157)
What's cool about these statistics-based approaches is that you mostly don't even need fully up-to-date statistics, just overall decent stats, unless you have an insane amount of churn. Meaning - you can get query speedup without insertion overhead: you choose to take that overhead any time you want using ANALYZE.
There's a fair amount of overlap, but where the databricks article explains the techniques with charts and high level explanations, I go over the code instead.
It basically shows that most cost-based optimizers are pretty bad at cardinality estimation, which compounds when queries use more joins.
Not that this isn't a great development in itself...
Very neat stuff from the databricks team!