It has parallels everywhere. How can you stop people poisoning food in a shop; walking into a school and stabbing a bunch of young kids; irradiating coins; deliberately driving cars into people; spreading sexual diseases; closing organisations with bomb threats etc. There's plenty of things you could do with practically zero chance of getting caught (and not everyone is worried about that) and for a low cost. You can't run a society that has a 100% success rate of preventing these sorts of things; you have to trust that they won't do it.

to your point: chasing terrorism is like trying to solve for all these (and many more) nefarious ways in which society isn't looking for wrongdoing. It's basically impossible to safeguard against every type of threat. Manage to lock down one sufficiently so that it's no longer attractive to accomplish ${evil_stuff}? Move on to the next one. We now see people getting run down with large trucks, since air travel is much more restricted. Everyone still suffers from restricted air travel, but will the TSA start inspecting every car driving over a bridge or an on-ramp now to prevent these new attack vectors? I sure hope not.

We swiped an idea from launching Nuclear Missiles: Typically you need two keys from two people to launch.

For really important accounts - we have three people who each know two thirds of the password. It requires two people to then log in and do damage.

For example if the root password was CatDogFish then

Person1: CatDog_

Person2: _DogFish

Person3: Cat_Fish

Two people can then log in and watch what the other person is doing.

This is like an informal version of Shamir's Secret Sharing

https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing

Maybe someone should implement a PAM module or something that requires authenticating as two different users with sudo privileges, so you can implement this for as many users as you like and let them each have a single password, instead of half of a password for each possible pair of users.

Use something with 2FA like e.g. https://duo.com/docs/duounix and give person A the password and person B the 2nd factor (phone with this Duo profile configured or whatever).

That doesn't eliminate the need to set up an account & password for every possible pair of users, rather than just every user.

If would also have to do session sharing so both people can watch each other and see what commands they run.

What if one of the people that has part of the key gets mad and leaves without sharing it?

https://blog.threatstack.com/who-watches-the-watchmen-securi... // https://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%... this idea has been around for a very long time.

Then you still have two users with a complete password set, from which you can either reconstruct the third permutation or generate a new set.

That only sort of helps. Do both people also watch each other while they run commands?

Because if not, once you are admin, you can install programs that let you become admin again at will.

Yes! We swiped the idea from the military. In certain dangerous places, they require two people to be there or nobody. One person is not allowed.

https://en.wikipedia.org/wiki/Two-man_rule

Well whoever set that up knows the whole password right?

Nope. It could be system generated and disseminated over secure channels.

Password rotation becomes necessary, and a little bit riskier, because now you have to deal with accidental lock-outs in a sane, coordinated manner.

and who implements the system that generates and disseminates over secure channel?

You are stuck trusting somebody, no matter what. Its turtles all the way down.

Eh, 3 pieces of a randomly generated string... hashed in memory, and only the result of the hash function is stored. Pretty basic concept.

Soon, dear throwaway, you'll be telling me we should live in fear of the locksmiths, for all their key blanks and such fiendish metal files to abrade them with. What if they should file down a butter knife into the shape of my precious bicycle chain's key?

I didn't say you should be afraid, simply that trying to build a system that doesn't require you to trust people is a fools errand.

So one person runs a keylogger and captures the full credentials?

But it's not a bad system.

Since the title says "Ex-Admin", revoking credentials when someone leaves the company is a must. If there's centralized auth (AD), this is pretty straightforward. If not, then at the very least it makes sense to use Puppet, Ansible, etc. to lock that person's account on all production servers.

Beyond that, be sure to keep regular backups (and test them), and audit all user actions. (feed the logs into something like Splunk, running on a separate machine)

I bet the ex-admin was also responsible for keeping backups and auditing/logging all access, in addition to managing pretty much everything else.

There really should be more systems that require 2 keys to do things like delete servers, load balancers, etc. I'm not aware of any. It's crazy to think that if an admins AWS/GCP/Azure account is compromised (or the cloud provider is compromised) a few commands/clicks to an API server could delete everything.

You prevent this by really trusting people (Trust, but verify). Don't screw them over. Treat them better than fairly. Give your employees raises before you give yourself anything.

And do backups. And then backups of those backups.

Yeah, the key is inspiring loyalty and creating a nurturing environment. I loved my first two employers and would go in to either, today, if something I worked on was broken and they couldn't fix it.

That's probably because when my grandmother died, my boss at QueBIT said "Ok, go home, call me when you can work again - however long that takes." There was never a discussion of PTO/HR policy, just human treatment.

Also remember to test restoring your backups or they don't count.

As crappy a job as it is to think about, start planning now. Have a checklist of everything everyone has access to. Have a procedure in place for what happens upon termination.

I worked retail to pay for college. Could always tell when a manager was getting the boot; they'd order new cylinders for all the doors. You kind of have to have that plan in place in IT too.

There should be a backup device or system offsite somewhere that automatically grabs incrementals of everything (and yells at you if it can't for some reason), essentially nobody should have any kind of remote access to it, and the fewest possible people should have physical access to it and gaining physical access should be such a hassle that you don't do it unless it's critical.

That's how I do it, anyway.

We can grant that this can be logistically difficult at certain scales, but it doesn't fall into the "engineering-impossible" bucket until you reach Facebook's size.

What you're describing here would require the company to be able to access their clients servers and data.

I don't know of any server provider (bare metal or cloud) that forces users to allow the company full access to their data (outside of managed providers were you voluntarily give this up , as you're paying them to fully manage your server)

If you are in AWS, you can set the employee's account permissions to not be able to delete servers or drives. In AWS you can also us CodeCommit and set permissions so that the Repos can't be deleted either.

If the CI/CD is done right, then no DevOps staff has any access to any servers and no one can delete anything except a scripts and AWS configurations.

The whole problem with limiting permissions is that you have to do all the work of deleting files, servers and drives.

If this was truly a former employee (wasn't made 'former' after the incident occurred) then all of their accounts should have been blocked from having access. Unless the guy built a back-door or something. IT and Network team may have dropped the ball on that one.

Usual DR stuff covers this and everything else, no one person should have the keys to the kingdom or in this case access to the offsite backups. Offsite because fire, earthquake, zombie apocalypse, covfefe,... google 'disaster recovery'. If you are a small company take the backups home with you and lock them in the safe in the den. Trust nobody including yourself.

  covfefe

So THAT'S what it means!