GitLab Database Incident – Live Report

Re:GitLab Database Incident – Live Report by throwaway8124 on Wednesday February 01, 2017 @07:03AM (#13540038)

Indeed. That's happened on our systems as well. Someone issued a reboot on what they thought was a non-critical host, and instead they did it on some very essential host. The host came up in a bad state, and critical services did not start (kerberos....).

It lead to a "Very Bad Day". I found out about it after reading the post mortem.

[ Parent | on HN ]

Re:GitLab Database Incident – Live Report by lovehashbrowns on Wednesday February 01, 2017 @01:39PM (#13543524)

One day, I almost accidentally shut off our master DB so I could update it. It's funny.. I leave these types of tasks for later in the day because I'm going to be tired and they're easy & quick tasks. But that almost backfired on me that day; I read the hostname a few times before it fully hit me that I wasn't where I was supposed to be.

[ Parent | on HN ]

Re:GitLab Database Incident – Live Report by foota on Wednesday February 01, 2017 @01:46PM (#13543613)

I always set my tmux window name to "LIVE" or something similar when I ssh into a live host.

[ Parent | on HN ]