News for hackers. Stuff that matters.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by phprecovery on Friday September 05, 2014 @04:15PM (#8276043)
The files are scanned TIFFs in the 25-30 MB range so we're thinking 5 seconds might be on the optimistic side.

Yes, definitely planning to parallelize but we're likely limited to four simultaneous instances so that will still take ~20-30 days.

Re:Ask HN: Any ideas on how to thumbnail 2M images? by meryn on Friday September 05, 2014 @04:43PM (#8276179)
I'm not sure what resolution these images are, but if it's very high-res, it may make sense to make intermediate-sized versions first. I expect that would make recreating thumbnails (i.e. in a different size - or multiple sizes) much quicker.

Also, converting the (hi-res) images to JPEG before further processing may make the data somewhat easier to deal with. 25-30MB per image * 2 million would be 50-60 terabytes of data? Plus, how often would someone really benefit from having access to a 25-30MB TIFF vs a much smaller JPEG version in the same resolution? That alone could be reason to start with JPEG compression first.

Re:Ask HN: Any ideas on how to thumbnail 2M images? by snowwrestler on Friday September 05, 2014 @04:52PM (#8276226)
I guess it's worth asking, why not just take 30 days to do it? Just work on something else until it's done.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by Igglyboo on Friday September 05, 2014 @04:26PM (#8276097)
Are cloud providers an option? You could use AWS of Google Compute Engine, setup 150 instances and do it in ~1 day.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by bilalhusain on Friday September 05, 2014 @04:30PM (#8276128)
As jenkstom pointed out, there might be bandwidth issues. I guess it'll require uploading few dozen TB data.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by snowwrestler on Friday September 05, 2014 @04:49PM (#8276214)
AWS has a "ship a hard drive" upload option.