News for hackers. Stuff that matters.
Re:GitHub Outage by chuhnk on Wednesday January 27, 2016 @07:52PM (#10985046)
I once ran rm -rf in the production mysql data directory.

Shit happens.

Re:GitHub Outage by zymhan on Wednesday January 27, 2016 @08:21PM (#10985280)
I did that too. Destroyed out Zabbix database. Neither that Zabbix server, nor the other one monitoring the server I destroyed, could alert us that anything had gone wrong for over an hour. I finally realized it when I couldn't login...

I was able to painstakingly rebuild the server after 9 hours without anyone noticing. To this day one of my biggest fuck ups and prouder accomplishments.

Re:GitHub Outage by postila on Wednesday January 27, 2016 @09:12PM (#10985597)
Are you saying that once you break something, you should break your monitoring as well, but do it very quickly since it may be too late? :)
Re:GitHub Outage by zymhan on Thursday January 28, 2016 @10:23AM (#10988654)
Yeah, if you're going to break your monitoring solution, you'd better shoot it in the head and vaporize the body.
Re:GitHub Outage by ivthreadp110 on Wednesday January 27, 2016 @08:03PM (#10985135)
When I worked at an ecommerce company the boss (who also was a programmer) did that too (on our master engine- not just website orders, but ebay, amazon, everything historical, inventory, RMAs.... etc etc... OOPS indeed... Yep- shit happens. *Recovered from it- was a hastle and shut down everything, but it was one of thoes whoops moments. Happens to the best of us.
Re:GitHub Outage by AdamJacobMuller on Wednesday January 27, 2016 @10:14PM (#10985912)
Been there. Done that. No backups.

Recovered all the data by using open file handles in /proc/.

Not a fun two hours.

Shit happens. Live and learn.

Re:GitHub Outage by marco-lavag on Wednesday January 27, 2016 @10:57PM (#10986129)
Have you written a blog post about that /proc/ trick? sounds like an interesting read
Re:GitHub Outage by teraflop on Wednesday January 27, 2016 @11:25PM (#10986269)
Here's an article that explains it: http://archive09.linux.com/feature/58142
Re:GitHub Outage by AdamJacobMuller on Thursday January 28, 2016 @01:04AM (#10986604)
I don't blog, but, what teraflop posted is basically correct.
Re:GitHub Outage by Shizza on Wednesday January 27, 2016 @09:46PM (#10985776)
I'm going to put that on a T-shirt
Re:GitHub Outage by thusoy on Friday February 05, 2016 @12:43AM (#11039756)
Re:GitHub Outage by cflat on Wednesday January 27, 2016 @09:43PM (#10985752)
You are the human manifestation of "the chaos monkey"
Re:GitHub Outage by thusoy on Friday February 05, 2016 @12:43AM (#11039753)
For anyone that feel they might fall under the same category (I do!), represent: https://teespring.com/human-chaos-monkey