News for hackers. Stuff that matters.
Re:Parallelizing Jobs with xargs by mattj on Friday March 13, 2009 @01:26AM (#514547)
Running these two commands in series is likely vastly overstating the performance gains - almost all your time is going to be spent in io, and the second time around you'll have a good chunk (if not all) of it in disk cache. Try running both a few repeated times and see if you enjoy the same gains (on my iphone right now, so I can't do this myself)
Re:Parallelizing Jobs with xargs by delano on Friday March 13, 2009 @04:29AM (#514643)
You are correct sir!

     $ time find . -newerct '10 hours ago' -print
     real	0m4.262s
     user	0m0.090s
     sys	0m1.014s

     $ time find . -newerct '10 hours ago' -print     
     real	0m0.516s
     user	0m0.057s
     sys	0m0.251s

     $ time find . -newerct '10 hours ago' -print     
     real	0m0.302s
     user	0m0.056s
     sys	0m0.244s

     $ time find . -newerct '10 hours ago' -print     
     real	0m0.311s
     user	0m0.057s
     sys	0m0.251s

     $ find . | wc -l
     13167
Is there a way to reliably flush the disk cache besides restarting?
Re:Parallelizing Jobs with xargs by jws on Friday March 13, 2009 @09:34AM (#514835)
Speaking of Linux, yes. You can echo mysterious numbers into /proc/sys/vm/drop_caches to flush various caches. http://jim.studt.net/depository/index.php/flushing-caches-fo... for a one liner, http://linux-mm.org/Drop_Caches for the info from the horse's mouth.

As an example, consider running md5sum on a directory of 1436 jpeg files of about 50k each on an Atom 330 ( 2 dies, 2 hyperthreads/die). The 8way numbers use xargs with -n8 and -P8. The flushed cache entries indicate that all three caches have been flushed immediately before the command ...

                    elapsed   user   system
     Cold cache:       11.98   0.56   1.75  ========================
     second run:        2.71   1.38   1.76  =====
      third run:        3.17   1.40   1.75  ======
  flushed cache:       12.11   0.58   1.80  ========================
  flushed cache, 8way: 12.64   1.00   2.27  =========================
     second run, 8way:  1.22   1.15   2.52  ==
      third run, 8way:  1.17   1.26   2.26  ==
I do not explain the oddity that user time seems abnormally low with flushed cache, or perhaps abnormally high with cache data present, particularly with the single threaded version.
Re:Parallelizing Jobs with xargs by delano on Friday March 13, 2009 @11:08AM (#514931)
Great, thanks.

What did you use to generate that output?

Re:Parallelizing Jobs with xargs by jws on Friday March 13, 2009 @03:08PM (#515198)
'time', and typing. The graph bars are my right index finger while I count in my head. I should probably patent that before it gets out. Wait, I just published. I'm screwed.
Re:Parallelizing Jobs with xargs by prewett on Friday March 13, 2009 @05:22PM (#515338)
You have a year after first publication to patent. But you have successfully prevented anyone else from patenting (unless they already applied).