News for hackers. Stuff that matters.

A comment on Command-line Tools can be 235x Faster than a Hadoop Cluster (2014). Parent comment

Re:Command-line Tools can be 235x Faster than a Hadoop Clust... by gaius on Thursday May 24, 2018 @03:33AM (#17141840)
The funny thing is we didn’t even consider our 50Tb to be particularly big, we knew guys running 200Tb...

BTW in R data.table is better for larger datasets than the default dataframe... or use the Revolution Analytics stuff on a server.

[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than a Hadoop Clust... by Groxx on Thursday May 24, 2018 @10:03PM (#17150082)
Oh yeah, the default dataframe is practically only for toy problems / very-already-filtered data (which accounts for a LOT of useful scenarios!). It's more that I've run into "big data" people who (╯°□°)╯︵ ┻━┻ when the default settings + dataframe choke, which is around 200MB of heap.
[ Parent | on HN ]