Re:GitLab Database Incident – Live Report
by atmosx on Wednesday February 01, 2017 @04:05AM
(#13539406)
The problem in this case was that YP confused the host. YP thought he was operating in db2 (the host that went out of sync) and not db1 (the host that held the data), a message that doesn't display the current host wouldn't help in this case.
[ Parent
| on HN ]
Re:GitLab Database Incident – Live Report
by throwaway8124 on Wednesday February 01, 2017 @07:03AM
(#13540038)
Indeed. That's happened on our systems as well. Someone issued a reboot on what they thought was a non-critical host, and instead they did it on some very essential host. The host came up in a bad state, and critical services did not start (kerberos....).
It lead to a "Very Bad Day". I found out about it after reading the post mortem.
Re:GitLab Database Incident – Live Report
by lovehashbrowns on Wednesday February 01, 2017 @01:39PM
(#13543524)
One day, I almost accidentally shut off our master DB so I could update it. It's funny.. I leave these types of tasks for later in the day because I'm going to be tired and they're easy & quick tasks. But that almost backfired on me that day; I read the hostname a few times before it fully hit me that I wasn't where I was supposed to be.