1. Company walks you through setting up FileVault (with key escrow) and VPN client, generating and uploading SSH public key immediately after unboxing laptop.
2. OneLogin + Duo (or pick your SSO/2FA scheme) for everything - internal webapps, GMail, etc.
3. SSH keys managed by Puppet.
4. 2FA verification of SSH logins (using pam-interactive), through a bastion (provided .ssh file makes it transparent), with OSX's default SSH agent, with pretty much all of those best practices configured except smartcards/certs.
5. Engineers have SSH access only to utility/development boxes. You can deploy code to production through a webapp (identifying the commit ID) but you can't get a shell as your application's user, and certainly not root.
6. Webapps moving behind a VPN.
In the rare case that you need to debug in production beyond what you can get from metrics/logs, you pair with a "blessed"/senior sysadmin type.
I <3 Duo Security.
Automate the build and deployment environment on a system that developers don't have direct access. Instead of pushing changes, the bot pulls a specified release branch, builds, tests, and deploys the code. All without human interaction.
If malicious code were somehow introduced from a developer's environment, it would be recorded and reflected in the commit history.
AFAIK, that's how GitHub manages deployments via HubBot. See https://www.youtube.com/watch?v=NST3u-GjjFw.
-----
To take things a step further, public-facing environments should be made immutable wherever possible. With the entire system being built and released as a whole. Docker alleviates some of the complexity and overhead but I think this space is where Unikernels have a lot of potential to shine.
There's a very good talk about how the Wunderlist team used chaos and frequent destruction of their envronments to overcome fear and uncertainty here. https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop.
Emphasis being placed on the the frequent disposal and recreation of environments rather than building long-running persistent environments.
This setup probably won't work for long-lived systems (ex databases). In those cases, access via a transient environment like a USB bootable OS would be ideal to prevent persistent viruses/trojans. Ironically, the best current options are security-focused distros like KaliLinux that put a special emphasis on avoiding persistent state. Maybe one day soon we'll see an admin-focused OS that better fits this role.
Most complex systems fail in unexpected ways, and the best way to debug is to grant the devs access to production machines. It can be temporary, monitored and through a secure channel, but it's still something most organizations can hardly live without.
So yeah, most organizations couldn't do this today, because they don't have the deploy process in place. But when an organization has made the switch, then it is much more realistic to assume that it can get along without granting ad hoc ssh access to production systems.
Hopefully, this practice -- and the tools required to make it happen -- get better with time.
It's also the least likely to happen. Attackers will have broken deep into your database due to poor webapp security before getting a hold of your ssh private key.
Not that SSH security doesn't matter. It's easy enough and carries enough risks that putting some effort into it is worthwhile.
A motivated attacker will always tend to try the "low tech" approaches first to rule them out as a first step because it it doesn't require as much sophisticated technical attention and in addition it's a well known fact that human beings are more often than not the weakest link in the chain of security.
It goes something like this:
1. enumerate the employees.. in other words build a list of all their work emails and try to obtain their personal emails and those of their spouses and children.
2. investigate the background of executives so that a compelling phishing email message can be crafted which looks like it originated from high up within the organization.
3. Send targeted phishing emails designed to bait people within the organization to visit webpages that exploit Adobe Flash attack vectors or other browser based vulnerabilities, or perhaps even lure them into installing a trojan directly.
4. after someone in the organization falls victim to the client-side attack, read their emails to learn more about the organization's structure so that your next phising attack can be more refined, specific, and compelling and can possibly utilize a real corporate email address.
5. Rinse and repeat until you eventually gain access to a developer laptop where you can grab production environment SSH keys. Now you own the network without even having to scan a single server and without leaving a trace in the log files.
For startups, the #1 risk is a vulnerability in some web app, or a forgotten admin panel, that leaks the entire database to an attacker. Not a complex attack conducted by a nation-sponsored offensive team.
But again, SSH security matters, you should take it seriously.
I think that you may have a dangerous attitude about it because in modern times it's not a question of whether or not you're a big enterprise or a startup it's a question of whether or not the dataset at the nucleus of your system would be valuable on the black market or not. If an attacker or group of attackers thinks that your dataset could be saleable one day in the future as your company continues to grow then instead of trying to buy your equity on the secondary markets they may invest in trying to "own" your infrastructure now before you become big enough to put your employees thru white-hat training around social engineering.
I suppose my point here is that it does make sense for startups to put their team through proper white-hat training but it doesn't have to be expensive because you can roll your own. What I suspect is that in 10 years or so this kind of anti-social engineering training will be a standard for any IT knowledge workers not just programmers and will likely be part of the job interview process.
We are aruging over something moot though, since we both agree.. take it seriously.
In my view there are emerging best practices in this area. There are two ways to reduce this risk and both are controversial:
1. Force developers to only develop software using an SSH terminal (by first connecting to a developer VPN via 2FA and then sshing into their secure development environment where may use tools like tmux, vi, and their programming language of choice to get the job done). In this scheme copying source codes or security credentials to a developer laptop is considered a violation and becomes a fireable offense no questions asked.
2. Require all developers to run a private USB-bootable linux desktop shell which is known to be clean. In this case they remain free to utilize modern desktop editors and code emulators (such as android simulator). It's even possible to setup secure persistence in these environments so that the developer's browser configuration, network/vpn config, dotfiles, apt installs, etc are stored on an encrypted filesystem on USB device. The reason why a USB image is preferred is because it's annoying to ask a new employee to repartiation her personal harddrive.
My suspicion is that as more of the tools developers need to rely on are cloud-based: (example: github, cloud9, jenkins, etc) we will eventually see these modern best practices against client-side attacks being adopted more broadly. The quality and reliability of hot-bootable ultra secure cloud operating systems has gone thru the roof over the past couple of years and I assume this trend will only accellerate due to the fact that Google Chrome OS continues to penerate more of the market and consumers are getting used to it.
TLDR: ssh keys existing on developer harddrives is an info-sec anti-pattern, they should only ever exist in system memory or in an encrypted partition on a USB stick.