Re:Parallel processing with Unix tools
by cat199 on Saturday September 23, 2017 @11:41AM
(#15320209)
For the pure SSH/SCP example there is PSSH/PSCP:
https://pypi.python.org/pypi/pssh/2.2
rdist I believe can operate in parallel as well.
Beyond this, devops tools e.g. ansible/puppet, etc. but they require a bit more or alot more setup, depending on the tool.
Re:Parallel processing with Unix tools
by cvwright on Saturday September 23, 2017 @04:13PM
(#15321479)
The pypi page for pssh links to an old (dead?) Google Code repository. Do you know if the project is still being maintained?
For most uses I find that ansible ad-hoc commands give a nice balance -- just a tiny bit of learning curve, and only a little bit different than typing the single SSH command that you want to run.
There’s also Rust Parallel[1]. It’s really fast, so it makes sense to use even for very quick things (but you might be running a lot so it adds up).
Overall it has become my preferred Parallel implementation.
Mr. Hess makes a lot of good points, most I think I agree with, so I can sympathize with his frustration.
The point about transferring files seems interesting. I haven't used gnu parallel in remote/ssh mode aside from simple tests, but that seems wicked useful for some situations. Is there a real and simple alternative to running gnu parallel on multiple remote hosts? Is that something the moreutils version can also do?