Re:Parallelizing Jobs with xargs
by mattj on Friday March 13, 2009 @01:26AM
(#514547)
Running these two commands in series is likely vastly overstating the performance gains - almost all your time is going to be spent in io, and the second time around you'll have a good chunk (if not all) of it in disk cache. Try running both a few repeated times and see if you enjoy the same gains (on my iphone right now, so I can't do this myself)
[ Parent
| on HN ]
You are correct sir!
$ time find . -newerct '10 hours ago' -print
real 0m4.262s
user 0m0.090s
sys 0m1.014s
$ time find . -newerct '10 hours ago' -print
real 0m0.516s
user 0m0.057s
sys 0m0.251s
$ time find . -newerct '10 hours ago' -print
real 0m0.302s
user 0m0.056s
sys 0m0.244s
$ time find . -newerct '10 hours ago' -print
real 0m0.311s
user 0m0.057s
sys 0m0.251s
$ find . | wc -l
13167
Is there a way to reliably flush the disk cache besides restarting?Speaking of Linux, yes. You can echo mysterious numbers into /proc/sys/vm/drop_caches to flush various caches. http://jim.studt.net/depository/index.php/flushing-caches-fo... for a one liner, http://linux-mm.org/Drop_Caches for the info from the horse's mouth.
As an example, consider running md5sum on a directory of 1436 jpeg files of about 50k each on an Atom 330 ( 2 dies, 2 hyperthreads/die). The 8way numbers use xargs with -n8 and -P8. The flushed cache entries indicate that all three caches have been flushed immediately before the command ...
elapsed user system
Cold cache: 11.98 0.56 1.75 ========================
second run: 2.71 1.38 1.76 =====
third run: 3.17 1.40 1.75 ======
flushed cache: 12.11 0.58 1.80 ========================
flushed cache, 8way: 12.64 1.00 2.27 =========================
second run, 8way: 1.22 1.15 2.52 ==
third run, 8way: 1.17 1.26 2.26 ==
I do not explain the oddity that user time seems abnormally low with flushed cache, or perhaps abnormally high with cache data present, particularly with the single threaded version.Great, thanks.
What did you use to generate that output?
'time', and typing. The graph bars are my right index finger while I count in my head. I should probably patent that before it gets out. Wait, I just published. I'm screwed.