Re:GNU Parallel - build and execute command lines from stand...
by paulitex on Sunday October 17, 2010 @04:39PM
(#1801353)
This is very cool, but a little opaque at first read... To get a quicker and digestible intro, watch the video introduction (linked on the page): http://www.youtube.com/watch?v=OpaiGYxkSuQ
[ Parent
| on HN ]
Re:GNU Parallel - build and execute command lines from stand...
by surki on Monday October 18, 2010 @03:08AM
(#1802630)
Well, the example(at least the first one) in that video is bit skewed. First, he runs gzip and then immediately runs 'parallel gzip' without dropping disk caches. So in the later case the bottleneck would be CPU rather than disk IO(everything read from disk cache in the RAM). IMO I expect for the work that is IO bound we won't see any significant improvement using parallel or anything similar.
Re:GNU Parallel - build and execute command lines from stand...
by oletange on Monday October 18, 2010 @07:32AM
(#1803022)
Ideas for next video are most welcome. The ideal task:
1. Is single threaded 2. Takes a lot of CPU 3. Is a task that everyone can understand and relate to and which is close to a real world scenario
I have loads of examples meeting requirement 1+2. It is 3 that is the hard part.
Post them to [email protected]
Re:GNU Parallel - build and execute command lines from stand...
by surki on Monday October 18, 2010 @09:01AM
(#1803278)
How about doing something with imagemagick or mencoder? I think video encoding/decoding gives a nice balance between disk and cpu usage.
Re:GNU Parallel - build and execute command lines from stand...
by eschulte on Sunday October 24, 2010 @04:00PM
(#1827087)
Here's an imagemagick example; over six minutes with xargs, under 20 seconds with parallel
$ ls *.png |wc -l
3580
$ time ls|sed 's/\(.*\)\..*/\1/'|parallel convert {}.png {}.ppm
ls --color 0.00s user 0.01s system 63% cpu 0.016 total
sed 's/\(.*\)\..*/\1/' 0.01s user 0.00s system 39% cpu 0.025 total
parallel convert {}.png {}.ppm 97.39s user 61.87s system 890% cpu 17.883 total
$ time ls|sed 's/\(.*\)\..*/\1/'|xargs -I {} convert {}.png {}.ppm
ls --color 0.01s user 0.00s system 63% cpu 0.016 total
sed 's/\(.*\)\..*/\1/' 0.01s user 0.00s system 39% cpu 0.025 total
xargs -I {} convert {}.png {}.ppm 93.08s user 47.88s system 38% cpu 6:10.88 total