Re:GNU Parallel 2018
by tbrock on Saturday August 17, 2019 @06:29PM
(#20726834)
xargs is great but doesn’t run the commands in parallel or chunk input does it?
[ Parent
| on HN ]
-P sets parallelism, -n number of arguments for each execution. Those are my main weapons. (-i is usually also necessary, and can allow for some pretty creative situations)
Re:GNU Parallel 2018
by hyperpallium on Sunday August 18, 2019 @12:41AM
(#20728273)
I don't swear on HN, but Holy Shit, I just added -P to xargs in a testsuite bash script, and it's 7 times faster.
I only have quad core, but -P values higher than 4 give better performance. Maybe because the tests take different amounts of time, and it avoids them backing up. To get x7, I had -P equal to the number of tests.
Re:GNU Parallel 2018
by waterhouse on Monday August 19, 2019 @02:43PM
(#20740506)
If the tests are ever waiting for I/O or something, or otherwise using less than one full CPU core, then I think that could explain why running more than the number of cores would be an improvement. If you run "top", or use the "time" utility, what fraction of a CPU core does each test use?
Re:GNU Parallel 2018
by hyperpallium on Tuesday August 20, 2019 @05:48AM
(#20745156)
Good thinking, but they don't do any file io, and I'm not sure how it could use less than a full CPU (wouldn't it just finish sooner? Though waiting for io could do that). OTOH it's java (on android, so dex format) and does some classloading of user classes... so maybe that is io? Here's time results:
real 0m1.818s
user 0m2.020s
sys 0m0.900s
top. (It completes in 1 sec, so put in a while loop - BTW top isn't behaving normally, "1" not giving multiple CPUs and not fitting on screen). Jus the top line: User 51%, System 23%, IOW 0%, IRQ 0%
Re:GNU Parallel 2018
by icebraining on Saturday August 17, 2019 @07:45PM
(#20727169)
It does. What xargs can't do is run stuff in parallel across many machines.
Yeah, you'd have to combine it with something like
Or pssh
It can, check the -n and -P arguments.
Re:GNU Parallel 2018
by gkfasdfasdf on Saturday August 17, 2019 @06:36PM
(#20726877)
xargs can do both and it rocks. https://www.google.com/search?q=xargs+parallel