Re:Does my data fit in RAM?
by nikhizzle on Wednesday February 12, 2020 @01:10PM
(#22311116)
You should use the cache or persist call on the spark data frame/dataset. Persist gives you more control.
[ Parent
| on HN ]
This is what most people in my org do, this is orders of magnitude slower than running queries on the same dataset on redshift with optimized presorting and distribution. Redshift doesn't scale for tens or hundreds of parallel users though, so looking for options
I'm about 75% joking: restore a new cluster from a snapshot for horizontal scaling. They did launch a feature along those lines, because my joke suggestion probably doesn't scale organizationally: take a look at Concurrency Scaling. This is also a fundamental feature of Snowflake with the separation of compute and storage.