News for hackers. Stuff that matters.
Re:Batch Processing Millions and Millions of Images by lars512 on Saturday July 10, 2010 @02:27AM (#1503180)
The problem of resizing all these images is an "embarrassingly parallel" one, right? You don't care about how fast any individual image is resized, only how fast they're resized in aggregate, and each image is a nice small chunk of work.

The author spends time tuning the number of workers and the number of OpenMP processes per worker for GraphicsMagick on his 16-core machines. Isn't this type of tuning a waste of time? Even using just two cores instead of one seems to introduce inefficiency. Wouldn't he have been better off just using 16 workers, each compiled without OpenMP so that it would run serially (and more efficiently)?

Re:Batch Processing Millions and Millions of Images by mahmud on Saturday July 10, 2010 @04:45AM (#1503374)
You really nailed it. Yes, yes and yes!

This is the sort of stupid batch task that can be done with a shell-script using all the machines in the office after hours. Cut out the middle ware, cut down the threads and fancy IPC, and hunker down with good ole minimal number of processes on a bunch of machines.

Heck, they might even see a big win if they use a different "server" memory management algorithm that's more tailored to batch, not-so-responsive applications.

Re:Batch Processing Millions and Millions of Images by gaius on Saturday July 10, 2010 @11:00AM (#1503725)
Barely even a shell script; you could do it in a one-liner. Let's say you had 16000 images and 16 cores,

    $ cat filelist.txt|xargs -n 1000 -P 16 ./myconvertprog
I do "batch" compression like this all the time.
Re:Batch Processing Millions and Millions of Images by mleonhard on Sunday July 11, 2010 @11:25AM (#1505600)
Thanks for pointing out the '-P' argument for xargs!
Re:Batch Processing Millions and Millions of Images by tange on Sunday July 11, 2010 @06:40AM (#1505315)
Like this (if you have GNU Parallel http://www.gnu.org/software/parallel/ installed) to run on multiple computers:

find images | parallel -j+0 –trc {.}.out -S server1,server2,: “do_stuff {} > {.}.out”

Watch the intro video for GNU Parallel: http://www.youtube.com/watch?v=OpaiGYxkSuQ