News for hackers. Stuff that matters.

A comment on Cold Showers. Parent comment

Re:Cold Showers by nostrademons on Saturday June 18, 2022 @07:52PM (#31794864)
My rule-of-thumb is that if you have less than a terabyte of data, you're better off processing it locally, and even that is pretty conservative. Big data is for when you have problem sets that simply will not fit on a single machine. With 4TB hard drives going for about $60, a lot of problems are better solved by simple algorithms in efficient programming languages on a single box.

There are some data sets where you really do need big-data tools, but it's for when you have petabyte-scale data, not megabyte/gigabyte-scale data.

[ Parent | on HN ]
Re:Cold Showers by atty on Saturday June 18, 2022 @10:08PM (#31795494)
Also depends on the complexity of the algorithm (specifically thinking of large neural networks). We have a model that requires 8 A100s for training due to the size of the activations. No way to replicate that on a local machine and have it train successfully in any reasonable time frame.

However the complexity of the algorithm many times scales with the size of the dataset, either the full corpus or the size of individual examples.

[ Parent | on HN ]