Re:Practical advice for analysis of large, complex data sets
by sn9 on Wednesday November 09, 2016 @04:51PM
(#12915044)
There's also this [0] on how to outperform Hadoop with command line tools.
Obviously, you should look at the role I/O costs can play. A lot of RAM and some SSD drives might be a better idea than Hadoop at this scale.
[0] http://aadrake.com/command-line-tools-can-be-235x-faster-tha...
Re:Practical advice for analysis of large, complex data sets
by greendragon on Wednesday November 09, 2016 @07:16PM
(#12916333)
It remains true. 4TB now is like $100 too, and SSDs that size are around even if they're an order of magnitude more expensive. (And then there's Seagate's 60 TB SSD prototype, clearly the future is with SSDs.)
https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
TL;DR
> But my data is 100GB/500GB/1TB!
> A 2 terabyte hard drive costs $94.99, 4 terabytes is $169.99. Buy one and stick it in a desktop computer or server. Then install Postgres on it.