News for hackers. Stuff that matters.
Re:Practical advice for analysis of large, complex data sets by jakub_g on Wednesday November 09, 2016 @04:08PM (#12914675)
I only know Hadoop from reading about it on HN, and the blog post I remember the most is "Don't use Hadoop - your data isn't that big"

https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

TL;DR

> But my data is 100GB/500GB/1TB!

> A 2 terabyte hard drive costs $94.99, 4 terabytes is $169.99. Buy one and stick it in a desktop computer or server. Then install Postgres on it.

Re:Practical advice for analysis of large, complex data sets by sn9 on Wednesday November 09, 2016 @04:51PM (#12915044)
There's also this [0] on how to outperform Hadoop with command line tools.

Obviously, you should look at the role I/O costs can play. A lot of RAM and some SSD drives might be a better idea than Hadoop at this scale.

[0] http://aadrake.com/command-line-tools-can-be-235x-faster-tha...

Re:Practical advice for analysis of large, complex data sets by greendragon on Wednesday November 09, 2016 @07:16PM (#12916333)
It remains true. 4TB now is like $100 too, and SSDs that size are around even if they're an order of magnitude more expensive. (And then there's Seagate's 60 TB SSD prototype, clearly the future is with SSDs.)