Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by bsenftner on Tuesday March 08, 2022 @07:33AM (#30599180)

I'm curious what constitutes "big data" anymore. In an intermediate machine learning course, we train on nearly a petabyte of data using Google Colab and Jupyter Notebooks. Nobody discusses the size of the data requiring any special treatment due to its size... would not 95% of a petabyte be "big data"?

[ Parent | on HN ]

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by happymellon on Tuesday March 08, 2022 @08:00AM (#30599392)

Big data is a shifting concept as computers gain more storage and faster commodity processors.

My general rule of thumb is whether it is too big to put on my laptop. So greater than a couple of Tb's.

[ Parent | on HN ]

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by pilotneko on Tuesday March 08, 2022 @08:44AM (#30599757)

What course are you taking? Imagenet is only 150 GB, and Common Crawl is only 320 TB.

Big data is a moving target, but I’m comfortable defining it as data too large to fit in memory. Obviously, you can always get a bigger node, my rule is thumb is that if you need generators, you are working with big data.

[ Parent | on HN ]