jssjr

I appreciate the honest feedback. We could have done better communicating about the problem. We've been marking single host failures in the dashboard for affected users and using our status page to reflect things like platform and regional issues, but there's clearly a spot in the middle where the status we're communicating and actual user experience don't line up.

We've been adding a ton more hardware lately to stay ahead of capacity issues and as you would expect this means the volume of hardware-shaped failures has increased even though the overall failure probability has decreased. There's more we can do to help users avoid these issues, there's more we can do to speed up recovery, and there's more we can do to let you know when you're impacted.

All this feedback matters. We hear it even when we drop the ball communicating.

Another status update from GitHub: "We have migrated to an unaffected DNS provider. Some users may experience problems with cached results as the change propagates."

We're maintaining yellow status for the foreseeable future while the changes to our NS records propagate. If you have the ability to flush caches for your resolver, this may help restore access.

Latest status message: https://twitter.com/githubstatus/status/789565863649304576

I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact.

"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.

[x-posted on https://news.ycombinator.com/item?id=12759697 as well]

I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact.

"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.

Sure. Feel free to call it anycast then. I usually hear anycast routing used in the context of achieving failover or routing flows to the closest server/POP, but there is probably a more formal definition in an RFC that I'll be pointed to shortly. =)

We are using BGP to advertise prefixes for GLB inside the data center to route flows to the directors. In our case all of the nodes are not on the same subnet (or at least not guaranteed to be) which is one of the reasons why we chose to avoid solutions requiring multicast. I expect Joe and Theo will get into more details about that in a future post though.

Anycast usually implies traffic will be directed to the nearest node advertising that prefix. The GLB directors leverage ECMP which provides the ability to balance flows across many available paths.

A "whole" piece of software would require you to have made the same data center design decisions we've made at GitHub. While some of our choices are opinionated, I think you'll find the GLB architecture adheres to the unix philosophy of individual components each doing one thing well.

Either way, I hope the upcoming engineering-focused posts are interesting and informative! Developing GLB was a challenging engineering project and if open-sourcing it means other companies can benefit from our work and spend more time developing their products, then I'll consider that a success.

I think you're referring to the GitHub Engineering blog post [1] about our git storage tier. We [2] store your code on at least 3 servers, which is an improvement in many ways from our previous storage architecture. There are a lot of servers [3] powering things but not the millions it would require to give every customer three dedicated machines. Developing efficient solutions to problems is a requirement (and a fun challenge!) for anything at GitHub's scale.

[1] http://githubengineering.com/introducing-dgit/

[2] I'm a GitHubber. https://github.com/jssjr

[3] https://twitter.com/GitHubEng/status/730429227896463360

This is really great work. Do you have any plans to open source some (or all) of the code behind Silverton?

GitHub's physical infrastructure team doesn't dictate what technologies our engineers can run on our hardware. We are interested in providing reliable server resources in a easily consumable manner. If someone wants to provision hardware to run docker containers or similar, that's great!

We may eventually offer higher order infrastructure or platform services internally, but it's not our current focus.

It's here at CodeConf!

https://gist.github.com/jssjr/018717ddcb81b76e1829#file-img_...

We use collectd extensively and it is wonderful software. Brubeck and collectd do very different jobs.

We did not try out statsite, but it also looks great! Brubeck has been under development internally since October 2012. We're really excited to be able to share it with everyone.

Indeed!

This sounds like a great idea for a future blog post. Thanks for the suggestion!

There are two posts on the blog right now:

Workload Analysis with MySQL's Performance Schema

http://githubengineering.com/using-mysql-performance-schema-...

Browser Monitoring for GitHub.com

http://githubengineering.com/browser-monitoring-for-github-c...

http://gocurb.com/ has been doing exactly that for many years.