Re:Cost-Based Optimizer in Apache Spark 2.2
by makmanalp on Thursday August 31, 2017 @11:54AM
(#15141157)
What's cool about these statistics-based approaches is that you mostly don't even need fully up-to-date statistics, just overall decent stats, unless you have an insane amount of churn. Meaning - you can get query speedup without insertion overhead: you choose to take that overhead any time you want using ANALYZE.
Very neat stuff from the databricks team!