News for hackers. Stuff that matters.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by fps on Monday July 25, 2016 @09:10AM (#12158387)
By implementing your own ssh, you appear to be ignoring lots of useful default ssh utilities, like ssh-agent and ~/.ssh/config eg.

    [derf@pan][09:06:20]-[~] $ orgalorg -o pandora -C uptime
    2016-07-25 09:06:28 [FATAL] can't create runner factory
    └─ can't read SSH key: '/Users/derf/.ssh/id_rsa'
       └─ can't read SSH key from file
          └─ open /Users/derf/.ssh/id_rsa: no such file or directory
    [derf@pan][09:06:28]-[~] $ orgalorg -p -o pandora -C uptime
    Password:
    2016-07-25 09:06:36  [WARN] still connecting to address after 2s: [derf@pandora:22]
    2016-07-25 09:06:44 [FATAL] acquiring global cluster lock failed
    └─ connection to 1 of 1 nodes failed
       └─ can't connect to nodes
          └─ can't connect to address: [derf@pandora:22]
             └─ dial tcp 18.60.0.124:22: i/o timeout
    [derf@pan][09:06:45]-[~] $ ssh pandora uptime
     09:06:49 up 37 days, 23:45,  0 users,  load average: 1.18, 0.70, 0.33
Is the assumption that I'm going to set up entries in /etc/hosts for all my systems, or add them to my network's DNS? What about ssh proxy hosts? The fact that you require a single ssh key to live in .ssh/id_rsa is going to make this useless for anyone that does any advanced usage of SSH.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by wjoe on Monday July 25, 2016 @10:27AM (#12158893)
That's a shame. That does immediately rule out my use case of using it at work for working with multiple servers. I have all of our servers in ~/.ssh/config (populated from a script), which use a number of different ssh keys. I assumed the host names in the orgalorg examples would work with the ones defined in ~/.ssh/config
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by seletskiy on Monday July 25, 2016 @10:48AM (#12159056)
Parsing `ssh_config` is not implemented right now as well as multiple SSH keys for auth. However, it can be implemented easy.

BTW, I have not implemented it, because I think it's very bad approach in maintaining clusters, e.g. storing host names inside local ~/.ssh/config file. Hosts should have DNS records.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by mugsie on Monday July 25, 2016 @02:48PM (#12160685)
yeap, but I have a nice .ssh/config that does wildcard masking , and a few other things to make sure I can log into servers without remembering which SSH Key I need to use, and what username.

e.g. I use the following snippet for my dev build boxes - they are rebuilt all the time, but I need to login and check system state from time to time.

  Host 10.250.60.*
    UserKnownHostsFile /dev/null
    StrictHostKeyChecking no
    IdentityFile ~/.ssh/local-dev-key
    User build
If you are implementing multiple SSH Keys, remember that a -lot- few people will have keys on GPG Cards, YubiKeys and other devices, not just on disk.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by subway on Monday July 25, 2016 @12:25PM (#12159723)
But how does one earn hipster cred without re-implementing battle-tested tools in go?
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by seletskiy on Monday July 25, 2016 @11:39AM (#12159387)
> By implementing your own ssh, you appear to be ignoring lots of useful default ssh utilities, like ssh-agent

What other utilities I'm ignoring besides of `ssh-agent`? `ssh-agent` support is a pending feature and implementing it is quite simple. I will be glad to see PR.

> Is the assumption that I'm going to set up entries in /etc/hosts for all my systems, or add them to my network's DNS?

Yes, you should to add your hosts to DNS. Otherwise, it's not production grade (we're talking about production clusters, aren't we?).

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by lobster_johnson on Monday July 25, 2016 @01:06PM (#12159998)
Have to agree with parent, supporting ~/.ssh/config is an absolute requirement for any tool that wishes to "reimplement" SSH. I routinely use custom user names, ports and proxycommands, via wildcard host names.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by fps on Monday July 25, 2016 @12:06PM (#12159582)
SSH config supports lots of esoteric options, like setting non-standards ports, usernames and specialized routing (using Proxycommand options).

Three issues, for me, with adding hosts to DNS:

1) I work from several networks - work, home, tethered. I only control the DNS zone that is set up as my default DNS search domain on one of those networks.

2) I work on hosts in several external networks - they shouldn't all be in the same DNS search domain.

3) I work primarily with cloud hosted, ephemeral servers. Cattle, not pets. Right now I have a tool that adds hosts from my EC2 accounts to my ssh config file (https://github.com/fredsmith/aws-ssh-config.)

I also have solved this problem using 21 lines of bash in my bashrc. (https://github.com/fredsmith/dotfiles/blob/master/bash/dsh) Mine doesn't have quite the level of polish of yours, but since it uses the system ssh binary, it handles special ssh configurations and seamlessly uses ssh-agent. I've used this on hundreds of hosts simultaneously, and it executes in parallel very quickly.

    [derf@pan][12:01:01]-[~] $ dsh smith.bz uptime
    pandora.smith.bz: 12:01:09 up 38 days, 2:40, 0 users, load average: 0.30, 0.23, 0.23
    dione.smith.bz: 12:01:11 up 42 days, 14:22, 2 users, load average: 0.00, 0.01, 0.05
    tethys.smith.bz: 12:01:11 up 23 days, 3:26, 1 user, load average: 0.00, 0.01, 0.05
    hyperion.smith.bz: 12:01:12 up 23 days, 3:01, 1 user, load average: 0.89, 0.87, 0.76
    io.smith.bz: 12:01:12 up 23 days, 3:26, 1 user, load average: 0.38, 0.73, 1.73
    janus.smith.bz: 12:01:12 up 31 days, 3 min, 1 user, load average: 0.37, 0.37, 0.28
    rhea.smith.bz: 12:01:12 up 42 days, 14:29, 2 users, load average: 0.03, 0.05, 0.07
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by ranman on Monday July 25, 2016 @04:00PM (#12161179)
https://aws.amazon.com/ec2/run-command/ Might be useful if you're doing a lot with AWS
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by brightball on Monday July 25, 2016 @12:34PM (#12159795)
SSH::Batch from CPAN is a handy and simple way to get s lot of what you're asking for.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by rdslw on Monday July 25, 2016 @09:47AM (#12158617)
Don't forget about fabric http://www.fabfile.org/ Older, much more tested, pythonic blah blah blah

DisclaimerAsImAmericanAndIDisclaimEverything: I'm not author. Not even user. I just know it exists.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by nine_k on Monday July 25, 2016 @11:48AM (#12159454)
Ansible is also tool that requires only ssh on the other end.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by TkTech on Monday July 25, 2016 @01:58PM (#12160356)
Small side note, both Ansible and Fabric build on top of Paramiko, a thorough and mature python library for SSH clients and servers.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by hobo_mark on Monday July 25, 2016 @08:29AM (#12158165)
For the 'Alternatives' section, there's also gnu parallel which I have been fairly happy with.

https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by seletskiy on Monday July 25, 2016 @08:39AM (#12158218)
I will add, thanks. But it uses binary SSH, it's not efficient.

Can you give me an example of such invokation? Layman's `parallel -S host1,host2 hosname` didn't work.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by BlackFingolfin on Monday July 25, 2016 @09:07AM (#12158358)
I am not quite sure what "it uses binary SSH" is meant to say. It sounds as if you were refering to the SSH protocol (which of course is "binary"), though my guess is that you mean to say that it relies on executing a separate "ssh binary" (note in particular "ssh" vs. "SSH"). If the latter interpretation is right, perhaps rephrase what you wrote in your README?
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by markuman on Monday July 25, 2016 @08:48AM (#12158261)
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by olet on Monday July 25, 2016 @09:33AM (#12158533)
parallel --nonall -S hosta,hostb --tag 'echo this is run on; hostname'
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by seletskiy on Monday July 25, 2016 @11:33AM (#12159342)
Change `echo this in ...` to `tail -f /var/log/mail.log` and parallel will show no output. orgalorg will output exactly what you expect.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by ole_tange on Monday July 25, 2016 @01:00PM (#12159970)
Use --line-buffer to get line buffered output.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by seletskiy on Monday July 25, 2016 @01:26PM (#12160124)
And it will never get past first server's output, because command is never finished.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by ole_tange on Monday July 25, 2016 @02:13PM (#12160443)
Ahh, but Sir, I think thou did not test it. Otherwise thou would have written differently.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by lobster_johnson on Monday July 25, 2016 @01:07PM (#12160003)
pssh (parallel SSH) is also very nice: http://linux.die.net/man/1/pssh.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by perlgeek on Monday July 25, 2016 @09:02AM (#12158332)
This looks like a really nice tool.

Also +1 for listing some alternatives and comparisons in the README, so that we don't have to bitch "but you should use $X instead" here :-)

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by BlackFingolfin on Monday July 25, 2016 @09:12AM (#12158404)
Well, having the list is certainly nice, though I don't really see a fair comparison there, more a list of "reasons why I think those other tools suck" ... :-). Some links to those external projects would also be nice. Just saying -- of course the author has probably better things to do than provide a link list to alternative software, but if you go to the trouble of making such a list, that might be handy.

And of course there are more alternatives out there, e.g. it perhaps Fabric <http://www.fabfile.org/> should also be in the list?

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by NickBusey on Monday July 25, 2016 @10:57AM (#12159104)
Though the reasons listed for what Orgalorg can do but Ansible can't are incorrect. Ansible can accept hosts lists from STDIN (-i '192.168.50.1,192.168.50.2').
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by seletskiy on Monday July 25, 2016 @11:31AM (#12159329)
It's not STDIN. I mean `resolve-dns 'blah' | orgalorg -s ...`, where `resolve-dns 'blah'` returns matching hosts from infrastructure DNS.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by shagabutdinov on Monday July 25, 2016 @08:58AM (#12158303)
This tool found it's niche as there were no reliable way to do parallel jobs with just one command.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by ysleepy on Monday July 25, 2016 @10:44AM (#12159018)
hehe :)
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by riskable on Monday July 25, 2016 @09:42AM (#12158591)
Reminds me of one of my first Python projects:

https://pypi.python.org/pypi/sshpt

"SSH Power Tool, Run commands and copy files to multiple servers simultaneously WITHOUT requiring pre-shared authentication keys"

These days I figured everyone that wanted to do such things would use a tool like Ansible.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by davidmr on Monday July 25, 2016 @02:33PM (#12160581)
First of all, congratulations on getting this to a point where you can show it off. A lot of hard work clearly went into this, and it's can't be an easy decision to unleash the hordes of HN programmers knowing the inevitable "constructive criticism" that results. I think a lot of the people who ask "why not just use X" might seem like they miss the point of writing a project to do something you need done just for the fun of it, but I'm an optimist so I assume they're just interested.

I'm an HPC sysadmin, so I use tools like this (specifically pdsh+pdcp) every day. Automation is great and we use it for just about everything, but the nature of the beast is that there's a lot of ad-hoc work and testing that needs to be done.

Specifically with respect to pdsh, just because it appears to have been abandoned, that doesn't necessarily make it the wrong tool for the job. It's pretty-well feature-complete for everything I use it for (that said, it was designed precisely for my needs, which likely aren't yours).

Some minor practical features that would make it a worthy replacement for the venerable pdsh:

  * range selection in the hostname selection going beyond bash expansion, like 
    -o compute[0012-0034,1123,1156-1199]
  * taking hostnames from a file is a must
  * supporting libgenders would be amazing
Nice work! I'll try it out and file github issues.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by jabl on Thursday July 28, 2016 @02:52AM (#12178759)
Not sure why people think pdsh is abandoned [1]? Seems the latest commit was ~a week ago: https://github.com/grondo/pdsh

[1] I'll guess that the current home is hard to find, as a google search finds the old sourceforge.net or googlecode pages.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by ole_tange on Monday July 25, 2016 @02:00PM (#12160370)
The examples in the GIF-animation can be emulated by GNU Parallel using this:

To make the emulation easier, make a simple alias:

  alias par_emul="parallel -j0 --ssh 'sshpass ssh' --nonall --tag --linebuffer"
If you want to supply a password run:

  SSHPASS=`ssh-askpass`
or set the password directly:

  SSHPASS=P4$$w0rd!
If the above is set up you can then do:

  orgalorg -o frontend1 -o frontend2 -p -C uptime
  par_emul -S frontend1 -S frontend2 uptime

  orgalorg -o frontend1 -o frontend2 -p -C top -bid 1
  par_emul -S frontend1 -S frontend2 top -bid 1
                                                                                            
  orgalorg -o frontend1 -o frontend2 -p -er /tmp -n 'md5sum /tmp/bigfile' -S bigfile
  par_emul -S frontend1 -S frontend2 --basefile bigfile --workdir /tmp  md5sum /tmp/bigfile
There are a few differences:

* GNU Parallel normally uses 'ssh' and recommends using 'ssh-agent'.

* With 'sshpass' you only have to type the password once.

* GNU Parallel does not have progress indication for the file transfer.

* GNU Parallel uses /./ to indicate which part of the path should be copied to the remote side: foo/bar/./sub/dir/file will append sub/dir/file to --workdir on the remote side.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by 300 on Monday July 25, 2016 @08:34AM (#12158196)
I use Rex for this (http://rexify.org/).

For example:

rex -H "frontend[01..15]" -e "say run 'uptime'"

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by aylmao on Monday July 25, 2016 @11:33AM (#12159350)
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by sigil on Monday July 25, 2016 @11:17AM (#12159238)
What's the added value over xargs + ssh? Eg:

    echo host1 host2 |
    xargs -P2 -n1 -I% ssh % uptime
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by chaosfox on Monday July 25, 2016 @12:34PM (#12159796)
if its just a single command, parallel[1] works great:

    parallel --tag 'ssh {} uptime' ::: host{1..10}
1. https://www.gnu.org/software/parallel/
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by ole_tange on Monday July 25, 2016 @01:04PM (#12159990)
If it is more: define a function.

    myfunc() {
       echo stuff;
       # More stuff
    }
    export -f myfunc
    parallel -Shost{1..10} --tag --env myfunc --nonall  myfunc
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by ziikutv on Monday July 25, 2016 @08:33AM (#12158190)
While a lot of this functionality already exists, this tool is great in combining it all.

P.S: not a big fan of the name :p

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by yojex on Monday July 25, 2016 @08:52AM (#12158279)
For the curious, it's an Adventure Time reference (spoilers) - http://adventuretime.wikia.com/wiki/Orgalorg
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by anonbanker on Monday July 25, 2016 @12:24PM (#12159716)
Pretty much the last good episode of Adventure Time, as well. The shark jump happened shortly thereafter (begining of the next season).
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by ziikutv on Monday July 25, 2016 @08:56AM (#12158298)
Thanks for the anecdote!
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by NeonMaster on Monday July 25, 2016 @09:57AM (#12158674)
We built something similar to this a while back: https://github.com/Mojang/Sift, it's great to see more alternatives popping up, seems like a great tool :)
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by roblabla on Monday July 25, 2016 @11:13AM (#12159214)
Off-Topic: What's the shell/config you're using in the README gif ? It looks damn cool.
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by nine_k on Monday July 25, 2016 @11:50AM (#12159459)
Can't help but quote: "If you have to SSH to your production box, your automation has failed".

This tool could be a nice thing to tinker and experiment with a cluster setup, though.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by batbomb on Monday July 25, 2016 @09:40AM (#12158581)
You don't support GSSAPI?
Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by jamiesonbecker on Monday July 25, 2016 @12:00PM (#12159527)
It's always good to see people challenging existing solutions .. this is how innovation happens. But you need to mention other good alternatives like fabric, paramiko, etc to list of alternatives... and plain old shell scripting.

Some of the other options in orgalog seem like a combination of rsync, autossh (which has a heartbeat protocol built in), xargs, and plain old bash might be a better option.

For ad hoc jobs like this, this seems like how I'd go about synchronizing local and remote directories on multiple servers in parallel:

    for host in host1 host2 host3
    do
         rsync -rpae ssh /my/directory/ $host:/remote/directory/ &
    done
    wait
That will synchronize a local directory with a remote directory, in parallel, and it's readable and uses software you probably already have installed.

Or, if you want to limit parallelization to no more than 16 boxes simultaneously, use in combination with xargs (ie something like xargs --max-procs 16 -n 1 and call a short bash function with the given hostname). Orgalog doesn't seem to support limiting to n hosts at a time?

Here's how to run the example uptime on hosts in parallel, but with batch limits:

orgalog:

    orgalorg -o <host>... -x -C whoami
bash:

    echo host1 host2 ... | xargs -d' ' -P2 -n1 -I{}     ssh {} uptime

Another orgalog example:

    orgalorg -o <host>... -p -i ~/.ssh/id_rsa.pub -C tee -a ~/.ssh/authorized_keys
bash:

    echo host1 host2 ... | xargs -d' ' -P2 -n1 -I{} scp ~/.ssh/authorized_keys {}
    # or, Userify cloud or On Site for >1 users

Of course, I'd just use Userify[1] in a local installation for that (disclaimer: CTO) but you could just modify the rsync example above or just use my old cloudadmins script[2] if your needs are small.

This was obviously a big effort just looking at the dozens of go files! I don't want to disparage what was obviously a huge amount of work from someone who spent a lot of time. A cautionary note for anyone thinking about this: always focus on automating non-security-sensitive tasks first and try to use the smallest amount of magic necessary to do a trick. Rolling your own crypto is never a good idea.

Clearly the OP is trying a lot of new ideas and we should expect good things in the future.

1. https://userify.com

2. https://github.com/jamiesonbecker/cloudadmins

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by seletskiy on Monday July 25, 2016 @12:16PM (#12159657)
Of course I'm aware of `for` loops in shell.

But note, that your solution lacks: valid termination of commands by `CTRL-C`, aggregated progress indication, password authentication, various fail modes (like try to connect to all servers first), elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root), no integration into other unix-way tools.

Also, you will little bit stuck trying to run `tail -f` via xargs ssh.

So, that's the reason for writing production-grade tool.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by jamiesonbecker on Monday July 25, 2016 @12:32PM (#12159775)
It's clear that you've put a lot of thought into this!

If I was going for production grade as opposed to a quick ad hoc job, I'd probably automate with a script to handle failures and send logs somewhere (thus wouldn't need or want control-C or progress indicators)

----

* valid termination of commands by `CTRL-C`

Good point. I'd probably use pkill for that, if my ad hoc job went haywire :)

----

* aggregated progress indication

You are correct, this is missing in my solution (nicely aggregated, anyway) -- but again if I was going to do this in production (i.e., automated), I'd probably not be doing it like this anyway.. I'd write a bash script, check status codes, timeouts, etc.

For example, delivering sorted output via log files:

    for host in host1 host2 host3
    do
         rsync -rpae ssh /my/directory/ $host:/remote/directory/ > rsync-$host.log 2>rsync-host.err  &
    done

and to syslog (hopefully a remote loghost)

    for host in host1 host2 host3
    do
         rsync -rpae ssh /my/directory/ $host:/remote/directory/ | logger -t "myscript-$host" 2>&1  &
    done
(note: --tag parameters for logger vary between RHEL and Debian-based distributions.. see the man page for specifics.)

----

* password authentication

Please never do that. We don't even provide that as an option at Userify (https://userify.com, SSH key management). SSH is one of the few places where passwords are completely obsolete.

----

* various fail modes (like try to connect to all servers first)

That seems like a simple thing to accomplish; are you saying orgalog connects to all servers first before connecting to all servers again?

----

* elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root)

Sure, that's easy..

echo host1 host2 ... | xargs -d' ' -P2 -n1 -I{} ssh {} sudo uptime

If that doesn't work for you, you're probably using a version of red hat with a broken sudoers file (which goes back nearly ten years!) but they're fixing it in the next release:

https://bugzilla.redhat.com/show_bug.cgi?id=1020147

In the meantime, just comment out the Defaults requiretty line to get the same fix.

Also, for the tty-requiring solutions that you mention, SSH has a command-line switch already: ssh -t and/or ssh -tt. `man 1 ssh` for details.

Userify configures sudo on remote systems, btw. (SSH key management, https://userify.com, blatant plug: CTO)

----

* no integration into other unix-way tools

I don't quite understand what you are saying.. I'd suggest that it's completely integrated into other unix tools, because it uses them instead of inventing a monolithic framework.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by seletskiy on Monday July 25, 2016 @01:22PM (#12160100)
> That seems like a simple thing to accomplish; are you saying orgalog connects to all servers first before connecting to all servers again?

orgalorg has two modes: either it will fail and do nothing if even one server fails; or it will skip all errors and continue to do whatever it can with the rest. It allows you to run orgalorg on all set of nodes, but specify different keys and access only subset of servers.

> Please never do that [...] SSH is one of the few places where passwords are completely obsolete.

You simply making declarative statement there without any underlying arguments. Please never do that.

We have passwords as part of our computing infrastructures right now and there are lot of cases where you need to use passwords (e.g., legacy infrastructure). So, supporting valid way of vastly used authentication doesn't sound 'bad' for me.

>> * elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root) > Sure, that's easy..

I'm talking about rsync. Try to use non-root login whily syncing root-owned files. Orgalorg solves that trivially (just add `-x` flag and it's done).

Of course, I'm aware that key-based authentication is way superior than password based. I'm trying using SSH keys whenever possible and have distributed solution for syncing them across all cluster (https://github.com/reconquest/shadowd). I do not find GUI-based solutions to be productive in work, so I prefer to use dead simple tools which are fail-safe because of simplicity.

Finally,

> but again if I was going to do this in production (i.e., automated), I'd probably not be doing it like this anyway.. I'd write a bash script, check status codes, timeouts, etc.

... and you will write orgalorg in bash.

Re:Show HN: Orgalorg – modern parallel SSH, file sync and cl... by jamiesonbecker on Monday July 25, 2016 @11:32PM (#12163155)
>>>> * elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root) > Sure, that's easy..

>> I'm talking about rsync. Try to use non-root login whily syncing root-owned files. Orgalorg solves that trivially (just add `-x` flag and it's done).

Fair enough! FWIW, that's actually trivial with native rsync also:

    rsync --rsync-path="sudo rsync" (...)
That calls rsync as 'sudo rsync' on the remote system.