News for hackers. Stuff that matters.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by orr94 on Friday September 05, 2014 @04:09PM (#8276010)

    At 5 seconds an image using a tool like ImageMagick (which might be optimistic)
Are you sure ImageMagick would take that long? I haven't timed it, but I don't recall thumbnail creation with ImageMagick taking that long.

Also, if you can parallelize it, it won't take 115 days.

Re:Ask HN: Any ideas on how to thumbnail 2M images? by orr94 on Friday September 05, 2014 @04:17PM (#8276058)
FWIW, some benchmarks say GraphicsMagick is faster than ImageMagick. http://www.compsci.wm.edu/SciClone/documentation/software/gr...

I suggest actually timing some of this stuff before writing off the idea. If an image can be processed in 1 second, and you can parallelize it to 4 cores, some quick and dirty (and probably useless) math says 2 million images could be done in just under 6 days. I don't think that's too shabby for 2 million images.

Re:Ask HN: Any ideas on how to thumbnail 2M images? by Igglyboo on Friday September 05, 2014 @04:29PM (#8276116)
I'd assume that the bottle neck would be I/O if these images are ~30mb as the OP stated in the thread. I don't think you'd see 4x speedups from parallelizing.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by orr94 on Friday September 05, 2014 @04:30PM (#8276129)
Good point. All the more reason to actually measure and find the bottleneck.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by chippy on Friday September 05, 2014 @04:32PM (#8276137)
Would faster disks / SSD's help in this case?
Re:Ask HN: Any ideas on how to thumbnail 2M images? by morpher on Friday September 05, 2014 @08:45PM (#8277023)
Not if the data has to be read from the slow disks to get it onto the fast ones.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by phprecovery on Friday September 05, 2014 @04:15PM (#8276043)
The files are scanned TIFFs in the 25-30 MB range so we're thinking 5 seconds might be on the optimistic side.

Yes, definitely planning to parallelize but we're likely limited to four simultaneous instances so that will still take ~20-30 days.

Re:Ask HN: Any ideas on how to thumbnail 2M images? by meryn on Friday September 05, 2014 @04:43PM (#8276179)
I'm not sure what resolution these images are, but if it's very high-res, it may make sense to make intermediate-sized versions first. I expect that would make recreating thumbnails (i.e. in a different size - or multiple sizes) much quicker.

Also, converting the (hi-res) images to JPEG before further processing may make the data somewhat easier to deal with. 25-30MB per image * 2 million would be 50-60 terabytes of data? Plus, how often would someone really benefit from having access to a 25-30MB TIFF vs a much smaller JPEG version in the same resolution? That alone could be reason to start with JPEG compression first.

Re:Ask HN: Any ideas on how to thumbnail 2M images? by snowwrestler on Friday September 05, 2014 @04:52PM (#8276226)
I guess it's worth asking, why not just take 30 days to do it? Just work on something else until it's done.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by Igglyboo on Friday September 05, 2014 @04:26PM (#8276097)
Are cloud providers an option? You could use AWS of Google Compute Engine, setup 150 instances and do it in ~1 day.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by bilalhusain on Friday September 05, 2014 @04:30PM (#8276128)
As jenkstom pointed out, there might be bandwidth issues. I guess it'll require uploading few dozen TB data.
Re:Ask HN: Any ideas on how to thumbnail 2M images? by snowwrestler on Friday September 05, 2014 @04:49PM (#8276214)
AWS has a "ship a hard drive" upload option.