News for hackers. Stuff that matters.
Re:Does my data fit in RAM? by nikhizzle on Wednesday February 12, 2020 @01:10PM (#22311116)
You should use the cache or persist call on the spark data frame/dataset. Persist gives you more control.
Re:Does my data fit in RAM? by ramraj07 on Wednesday February 12, 2020 @02:05PM (#22311689)
This is what most people in my org do, this is orders of magnitude slower than running queries on the same dataset on redshift with optimized presorting and distribution. Redshift doesn't scale for tens or hundreds of parallel users though, so looking for options
Re:Does my data fit in RAM? by zten on Wednesday February 12, 2020 @03:15PM (#22312319)
I'm about 75% joking: restore a new cluster from a snapshot for horizontal scaling. They did launch a feature along those lines, because my joke suggestion probably doesn't scale organizationally: take a look at Concurrency Scaling. This is also a fundamental feature of Snowflake with the separation of compute and storage.
Re:Does my data fit in RAM? by ramraj07 on Wednesday February 12, 2020 @05:57PM (#22313713)
Exactly the same game plan for us now, evaluating snowflake, but wanted to check if there's a fundamentally different paradigm that can be much more faster and scalablr