Re:Batch Processing Millions and Millions of Images
by mikebrittain on Saturday July 10, 2010 @12:04PM
(#1503851)
I am informed by the NYT project. :) For practical reasons, that approach wasn't right for us... at the time. Those details (for sake of clarity and sanity) didn't make it to the article.
This system doesn't scale out without ad-hoc partitioning. I think Etsy's approach here should have been informed by the New York Times' project a few years ago. They converted 4 TB of scanned tiffs (their article archive) into PDFs on a hadoop cluster running on ec2 and s3. Parallelizing the process across hardware nodes is the scale free way to do this, exactly what hadoop was intended for.
All that said, I know there are smart folks at Etsy. I'll give 'em the benefit of the doubt; there may be a good reason not to go that route but this write up didn't make that clear.