Re:Fundamentals of Incident Management
by burnished on Monday August 09, 2021 @02:21PM
(#28120316)
You're mixed up, they're drilling. You drill so that when an emergency happens and it's 4:30am and you're bleary eyed your hands already know what to do (and are doing it) before your eyes even open all the way.
[ Parent
| on HN ]
Re:Fundamentals of Incident Management
by candiddevmike on Monday August 09, 2021 @03:23PM
(#28121185)
This sounds like a great way to wipe a database accidentally (like GitLab). The worst thing you can do to help fix a problem is having people asleep at the wheel.
Noted, but the point of the drill is precisely to uncover these failure modes and attempt to fix them. e.g. you might have automated runbooks to fix the problem rather than access the DB directly. You might have frequent backups and processes to easily restore from backups in case of database wipes.