Three issues, for me, with adding hosts to DNS:
1) I work from several networks - work, home, tethered. I only control the DNS zone that is set up as my default DNS search domain on one of those networks.
2) I work on hosts in several external networks - they shouldn't all be in the same DNS search domain.
3) I work primarily with cloud hosted, ephemeral servers. Cattle, not pets. Right now I have a tool that adds hosts from my EC2 accounts to my ssh config file (https://github.com/fredsmith/aws-ssh-config.)
I also have solved this problem using 21 lines of bash in my bashrc. (https://github.com/fredsmith/dotfiles/blob/master/bash/dsh) Mine doesn't have quite the level of polish of yours, but since it uses the system ssh binary, it handles special ssh configurations and seamlessly uses ssh-agent. I've used this on hundreds of hosts simultaneously, and it executes in parallel very quickly.
[derf@pan][12:01:01]-[~] $ dsh smith.bz uptime
pandora.smith.bz: 12:01:09 up 38 days, 2:40, 0 users, load average: 0.30, 0.23, 0.23
dione.smith.bz: 12:01:11 up 42 days, 14:22, 2 users, load average: 0.00, 0.01, 0.05
tethys.smith.bz: 12:01:11 up 23 days, 3:26, 1 user, load average: 0.00, 0.01, 0.05
hyperion.smith.bz: 12:01:12 up 23 days, 3:01, 1 user, load average: 0.89, 0.87, 0.76
io.smith.bz: 12:01:12 up 23 days, 3:26, 1 user, load average: 0.38, 0.73, 1.73
janus.smith.bz: 12:01:12 up 31 days, 3 min, 1 user, load average: 0.37, 0.37, 0.28
rhea.smith.bz: 12:01:12 up 42 days, 14:29, 2 users, load average: 0.03, 0.05, 0.07
What other utilities I'm ignoring besides of `ssh-agent`? `ssh-agent` support is a pending feature and implementing it is quite simple. I will be glad to see PR.
> Is the assumption that I'm going to set up entries in /etc/hosts for all my systems, or add them to my network's DNS?
Yes, you should to add your hosts to DNS. Otherwise, it's not production grade (we're talking about production clusters, aren't we?).