News for hackers. Stuff that matters.

A comment on Presto: Interacting with petabytes of data at Facebook. Parent comment

Re:Presto: Interacting with petabytes of data at Facebook by spikels on Wednesday November 06, 2013 @05:55PM (#6686147)
The tradeoffs are both complex and evolving and are different for each project/product. In general Hadoop is more scalable and more flexible but much slower.

For example as of today RedShift can hold a maximum of 256 terrabytes of compressed data while Facebook's Hadoop cluster was over 200 Petabytes in late 2012. RedShift only supports limited query and data types and a single index while Hadoop can theoretically handle arbitrary data processing. But if these constraints are acceptable then RedShift will likely be orders of magnitude faster in most cases.

Other projects/products will have different tradeoffs but they are almost always faster as this was almost always the primary goal.

[ Parent | on HN ]
Re:Presto: Interacting with petabytes of data at Facebook by cjg_ on Thursday November 07, 2013 @12:13PM (#6690605)
Where do you get the limit of 256TB compressed from?

     Amazon Redshift enables you to start with as little 
     as a single 2TB XL node and scale up all the way to
     a hundred 16TB 8XL nodes for 1.6PB of compressed user data.
from http://aws.amazon.com/redshift/features-and-benefits/
[ Parent | on HN ]
Re:Presto: Interacting with petabytes of data at Facebook by spikels on Thursday November 07, 2013 @04:53PM (#6692677)
My mistake, thank for the correction.

The 16 node limit I was familiar with is not a hard limit. You can request more nodes.

http://aws.amazon.com/redshift/faqs/#0080

It would be interesting to see performance comparisons on these huge datasets. I would expect to see new and interesting problems at that scale.

[ Parent | on HN ]