News for hackers. Stuff that matters.
Re:Batch Processing Millions and Millions of Images by spidaman on Saturday July 10, 2010 @11:17AM (#1503761)
Hate to be the Monday morning quarterback but hey, it's Saturday morning and I'm stuck in a Starbucks in Benicia. So, this is a nice walk through the optimization process but it is a fundamentally unscalable system. When the workload triples next year, does it make sense to scale up the hardware? (No)

This system doesn't scale out without ad-hoc partitioning. I think Etsy's approach here should have been informed by the New York Times' project a few years ago. They converted 4 TB of scanned tiffs (their article archive) into PDFs on a hadoop cluster running on ec2 and s3. Parallelizing the process across hardware nodes is the scale free way to do this, exactly what hadoop was intended for.

All that said, I know there are smart folks at Etsy. I'll give 'em the benefit of the doubt; there may be a good reason not to go that route but this write up didn't make that clear.

Re:Batch Processing Millions and Millions of Images by mikebrittain on Saturday July 10, 2010 @12:04PM (#1503851)
I am informed by the NYT project. :) For practical reasons, that approach wasn't right for us... at the time. Those details (for sake of clarity and sanity) didn't make it to the article.