Re:GitLab Database Incident – Live Report
by philjohn on Wednesday February 01, 2017 @05:13AM
(#13539620)
At my last place, the terminal on prod servers had a red background, HA servers was amber and staging servers green - worked a treat.
Re:GitLab Database Incident – Live Report
by foota on Wednesday February 01, 2017 @02:38AM
(#13539075)
Perhaps, Do you really want to delete the dir XYZ with 300000000000 bytes in it?
Re:GitLab Database Incident – Live Report
by atmosx on Wednesday February 01, 2017 @04:05AM
(#13539406)
The problem in this case was that YP confused the host. YP thought he was operating in db2 (the host that went out of sync) and not db1 (the host that held the data), a message that doesn't display the current host wouldn't help in this case.
Re:GitLab Database Incident – Live Report
by throwaway8124 on Wednesday February 01, 2017 @07:03AM
(#13540038)
Indeed. That's happened on our systems as well. Someone issued a reboot on what they thought was a non-critical host, and instead they did it on some very essential host. The host came up in a bad state, and critical services did not start (kerberos....).
It lead to a "Very Bad Day". I found out about it after reading the post mortem.
Re:GitLab Database Incident – Live Report
by lovehashbrowns on Wednesday February 01, 2017 @01:39PM
(#13543524)
One day, I almost accidentally shut off our master DB so I could update it. It's funny.. I leave these types of tasks for later in the day because I'm going to be tired and they're easy & quick tasks. But that almost backfired on me that day; I read the hostname a few times before it fully hit me that I wasn't where I was supposed to be.
The best thing to do is to never operate with 2 terminals simultaneously, when one of them is a production env, better login/logout or at least minimise it.