I did that too. Destroyed out Zabbix database. Neither that Zabbix server, nor the other one monitoring the server I destroyed, could alert us that anything had gone wrong for over an hour. I finally realized it when I couldn't login...
I was able to painstakingly rebuild the server after 9 hours without anyone noticing. To this day one of my biggest fuck ups and prouder accomplishments.
Are you saying that once you break something, you should break your monitoring as well, but do it very quickly since it may be too late? :)
Yeah, if you're going to break your monitoring solution, you'd better shoot it in the head and vaporize the body.
Re:GitHub Outage
by ivthreadp110 on Wednesday January 27, 2016 @08:03PM
(#10985135)
When I worked at an ecommerce company the boss (who also was a programmer) did that too (on our master engine- not just website orders, but ebay, amazon, everything historical, inventory, RMAs.... etc etc... OOPS indeed... Yep- shit happens. *Recovered from it- was a hastle and shut down everything, but it was one of thoes whoops moments. Happens to the best of us.
Re:GitHub Outage
by AdamJacobMuller on Wednesday January 27, 2016 @10:14PM
(#10985912)
Been there. Done that. No backups.
Recovered all the data by using open file handles in /proc/.
Not a fun two hours.
Shit happens. Live and learn.
Re:GitHub Outage
by marco-lavag on Wednesday January 27, 2016 @10:57PM
(#10986129)
Have you written a blog post about that /proc/ trick? sounds like an interesting read
Here's an article that explains it: http://archive09.linux.com/feature/58142
Re:GitHub Outage
by AdamJacobMuller on Thursday January 28, 2016 @01:04AM
(#10986604)
I don't blog, but, what teraflop posted is basically correct.
I'm going to put that on a T-shirt
This close enough? https://teespring.com/human-chaos-monkey
You are the human manifestation of "the chaos monkey"
For anyone that feel they might fall under the same category (I do!), represent: https://teespring.com/human-chaos-monkey
Shit happens.