News for hackers. Stuff that matters.
Re:GitHub Outage by ikawe on Wednesday January 27, 2016 @07:33PM (#10984849)
Sorry, I think I caused this. =[

    bower jquery#1.11.3                       not-cached git://github.com/jquery/jquery-dist.git#1.11.3
    bower jquery#1.11.3                          resolve git://github.com/jquery/jquery-dist.git#1.11.3
    bower foundation#~5.5.2                       cached git://github.com/zurb/bower-foundation.git#5.5.3
    bower foundation#~5.5.2                     validate 5.5.3 against git://github.com/zurb/bower-foundation.git#~5.5.2
    bower ember#^2.3.0                           ECMDERR Failed to execute "git ls-remote --tags --heads git://github.com/components/ember.git", exit code of #128 
    fatal: remote error:
Mid bower install. Rly srry guys!!! =[
Re:GitHub Outage by i336_ on Wednesday January 27, 2016 @08:12PM (#10985218)
So... you were doing a production redeploy, and it crashed?

The URL "git://github.com/components/ember.git" [1] suggests this is an internal GitHub Bower build log, but your post history doesn't mention anything about GitHub (let alone whether you work there), so I'm not 100.00% sure.

[1] https://webcache.googleusercontent.com/search?q=cache:3e00jl...

Assuming this is, in fact, a GH Bower log, the first thing that came to mind was that this architecture isn't (and possibly should be) using a dual-silo approach: when you upgrade, the upgrade gets loaded into a new blank namespace/environment, tested, and if it worked (passes CI test coverage or something like that), the main entry point is switched to the new environment (maybe with a web server restart or config rehash) and the old environment gets purged (possibly after a trial period). The current stack looks quite akin to "click this button to flash the new firmware and DO NOT UNPLUG your device or you'll brick it."

But then I realized... wait. You guys have like... isn't it like, a few dozen RoR worker boxes? Was this crash on the inbound router or something? xD

It's all good though; consider this "curious criticism" - like constructive criticism, but with extra sympathy. And hey, GH's never broken on my watch before (not that I need it atm... hopefully); this is interesting =P

Re:GitHub Outage by gjtorikian on Wednesday January 27, 2016 @08:59PM (#10985513)
Hate to be the bearer of bad news, but ikawe was joking about the bower thing.

Source: I work at GitHub.

Re:GitHub Outage by AdamJacobMuller on Wednesday January 27, 2016 @09:26PM (#10985673)
> Source: I work at GitHub.

So, you're saying you broke it.

Everybody, over here, gang up on this guy.

Re:GitHub Outage by ivthreadp110 on Wednesday January 27, 2016 @10:01PM (#10985857)
Shit happens some time. Who cares who broke it- what matters is they're fixing it. I hope github does make public a official reason tomorrow. Not looking to blame anyone- I just want to know what they're doing internally to make sure it does not happen again... adaptation. Even if it's from some external hack or ddos attack- how can they plan on building redundancies in. I am pretty sure GitHub's dev team is talented- I'm being such an anti-troll.
Re:GitHub Outage by AdamJacobMuller on Wednesday January 27, 2016 @10:12PM (#10985906)
My comment was quite joking in nature.

> I hope github does make public a official reason tomorrow

I do too, but, mostly I find post-mortems quite interesting to read.

If your services are at some 95% uptime or lower, you're doing something (serveral things) very wrong and its probably not that interesting to me.

Getting from 95% -> 99% you probably did some interesting things there.

Going into multiple .9s beyond that, you're likely doing quite a bit of interesting stuff but what I find more interesting is where you went wrong. Figuring out not just where you went wrong, but, what your incorrect assumptions were and WHY they were wrong. "We believed X could never fail because of Y, and even if X did fail, it would not cause production impact because of Z!"

Re:GitHub Outage by i336_ on Wednesday January 27, 2016 @11:31PM (#10986294)
Ah, glad I didn't presume it was a valid message! I've heard people joke about this kind of thing exactly this way in the past.

But now my "technical breakdown info" box has no tidbits in it. :P

I'm glad you're back up now (sortakinda - what sort of traffic are you sustaining right now? :D), but a rough idea of what asploded would be really cool to know about.

Speaking of which, I'd like to take a moment to make a strong point about the fact that disaster-recovery situations don't get blogged about enough. Vague "we fixed it" datapoints get buried in status update logs like it's something to hide and hope nobody brings up.

In situations like these, the only constructive perspective is for everyone to accept that something went horribly wrong and not make a fuss about it, and if such a mentality can be established, this creates an environment within which we can share technical breakdowns of "we found ourselves in XYZ position and then we did these thirty highly specific things in heroically record time to be up and running again", and I think sharing this type of info would potentially be more educational than setup tutorials or the "we switched to X and it improved Y by 1400%" type things the Net's full of. Sure, you'd have to generalize and probably give a lot of backstory about infrastructure, but it's becoming trendy (in a sense) for companies to describe their operations in precisely this way, so it's not completely nonviable.

(Note my use of the angle of "we found ourselves in XYZ position" - maybe a small highlight of what led up to the disaster would be included (worth considering if the information would be educational), maybe not. In a blog context, moderating comments to keep the discussion on-track and constructive may be necessary, but IMO would be worth it.)

Re:GitHub Outage by ivthreadp110 on Wednesday January 27, 2016 @09:01PM (#10985529)
I missed the joke. Either way- hope you get everything up and running soon!
Re:GitHub Outage by spiralganglion on Wednesday January 27, 2016 @08:58PM (#10985501)
Perhaps you missed the joke?
Re:GitHub Outage by snarkyturtle on Wednesday January 27, 2016 @08:03PM (#10985130)
That's what you get for using bower and jquery. The react/webpack gods are angry.
Re:GitHub Outage by ivthreadp110 on Wednesday January 27, 2016 @08:04PM (#10985147)
or the react/webpack gods are laughing...
Re:GitHub Outage by chuhnk on Wednesday January 27, 2016 @07:52PM (#10985046)
I once ran rm -rf in the production mysql data directory.

Shit happens.

Re:GitHub Outage by zymhan on Wednesday January 27, 2016 @08:21PM (#10985280)
I did that too. Destroyed out Zabbix database. Neither that Zabbix server, nor the other one monitoring the server I destroyed, could alert us that anything had gone wrong for over an hour. I finally realized it when I couldn't login...

I was able to painstakingly rebuild the server after 9 hours without anyone noticing. To this day one of my biggest fuck ups and prouder accomplishments.

Re:GitHub Outage by postila on Wednesday January 27, 2016 @09:12PM (#10985597)
Are you saying that once you break something, you should break your monitoring as well, but do it very quickly since it may be too late? :)
Re:GitHub Outage by zymhan on Thursday January 28, 2016 @10:23AM (#10988654)
Yeah, if you're going to break your monitoring solution, you'd better shoot it in the head and vaporize the body.
Re:GitHub Outage by ivthreadp110 on Wednesday January 27, 2016 @08:03PM (#10985135)
When I worked at an ecommerce company the boss (who also was a programmer) did that too (on our master engine- not just website orders, but ebay, amazon, everything historical, inventory, RMAs.... etc etc... OOPS indeed... Yep- shit happens. *Recovered from it- was a hastle and shut down everything, but it was one of thoes whoops moments. Happens to the best of us.
Re:GitHub Outage by AdamJacobMuller on Wednesday January 27, 2016 @10:14PM (#10985912)
Been there. Done that. No backups.

Recovered all the data by using open file handles in /proc/.

Not a fun two hours.

Shit happens. Live and learn.

Re:GitHub Outage by marco-lavag on Wednesday January 27, 2016 @10:57PM (#10986129)
Have you written a blog post about that /proc/ trick? sounds like an interesting read
Re:GitHub Outage by teraflop on Wednesday January 27, 2016 @11:25PM (#10986269)
Here's an article that explains it: http://archive09.linux.com/feature/58142
Re:GitHub Outage by AdamJacobMuller on Thursday January 28, 2016 @01:04AM (#10986604)
I don't blog, but, what teraflop posted is basically correct.
Re:GitHub Outage by Shizza on Wednesday January 27, 2016 @09:46PM (#10985776)
I'm going to put that on a T-shirt
Re:GitHub Outage by thusoy on Friday February 05, 2016 @12:43AM (#11039756)
Re:GitHub Outage by cflat on Wednesday January 27, 2016 @09:43PM (#10985752)
You are the human manifestation of "the chaos monkey"
Re:GitHub Outage by thusoy on Friday February 05, 2016 @12:43AM (#11039753)
For anyone that feel they might fall under the same category (I do!), represent: https://teespring.com/human-chaos-monkey
Re:GitHub Outage by vampolo on Wednesday January 27, 2016 @07:56PM (#10985078)
Oh! i see! So you are the guy that DDoSed github by downloading Ember!
Re:GitHub Outage by spiralganglion on Wednesday January 27, 2016 @08:02PM (#10985129)
I thought the point of heavy client-side frameworks was to take load off the server.
Re:GitHub Outage by rdoherty on Wednesday January 27, 2016 @07:56PM (#10985076)
Can someone explain this to people who don't use bower?
Re:GitHub Outage by StavrosK on Wednesday January 27, 2016 @07:58PM (#10985091)
He was just cloning stuff and it failed mid-way.
Re:GitHub Outage by borismus on Wednesday January 27, 2016 @08:08PM (#10985181)
Maybe by the time you switch to npm, github will come back up?
Re:GitHub Outage by hartator on Wednesday January 27, 2016 @07:41PM (#10984926)
Cool that you have time to update us. :)