most people use the term to mean 'a coder who can configure apache'
this whole incident reeks of a lack of OpSec and general Ops knowledge.
Lack of offsite backups is one thing, but if you're using cloud infrastructure you never place all eggs in one basket.
your instances have to be disposable and reproducible, at the first hint of a break in my default action is to torch and rebuild.
Or rather shutdown, preserve, start replacements. IMHO your automation should be able to deal with the old preserved instances being left alone.
There are some things you aren't going to expect (compromise of your AWS console). This could have been solved by having MFA enabled, as well as having the app push backups in realtime, versioned with delete protection, to S3 buckets under the control of another account (write access, but no delete access).
Show of hands how many people here are doing it this way.
I'm sorry, but this is a succession of things not to do in terms of system operations. Probably the team never managed mission critical platforms before, and hopefully they now learned the lesson.
Software people tend to make all kinds of assumptions about hardware that do not work out in practice.
And now that spinning up a couple of servers on AWS and creating snapshots on-the-fly are so easy, it gives the false impression that you don't need much to act as a sysadmin.
As for the rest of us: AWS is a great one-stop shop. Unfortunately, using just AWS puts you in the "all the eggs in one basket" scenario that we were warned against as children.
Sending a copy to Glacier is a second basket.
Does Amazon not have 30-day undelete for bulk storage? Seems crazy.
Not in my books. Recommended to be sure, but this has far too many single points of failure. To name a few:
- Software corrupts data
- Hardware corrupts data
- Social engineering bypasses 2FA
Two-factor authentication is a second basket is a better carrying strap but it's still on a single basket.
If you can delete it with a single control panel, it doesn't count as an offsite back. Fire the devops