Re:Launch HN: Data Mechanics (YC S19) – The Simplest Way to ...
by jstephan on Monday May 11, 2020 @01:10PM
(#23144414)
Thanks, great question !
Dynamic allocation is only enabled on our Spark 3.0 image (from the 3.0-preview branch, since the official 3.0 isn't released yet). It works by tracking which executors are storing active shuffle files. These executors will not be removed when downscaling. More info here: https://issues.apache.org/jira/browse/SPARK-27963
It's not perfect, but there are more improvements for dynamic allocation being worked on (remote shuffle service for Kubernetes).
Dynamic allocation is only enabled on our Spark 3.0 image (from the 3.0-preview branch, since the official 3.0 isn't released yet). It works by tracking which executors are storing active shuffle files. These executors will not be removed when downscaling. More info here: https://issues.apache.org/jira/browse/SPARK-27963
It's not perfect, but there are more improvements for dynamic allocation being worked on (remote shuffle service for Kubernetes).