Re:Status.github.com: “We're failing over a data storage sys...
by sciurus on Sunday October 21, 2018 @09:33PM
(#18271505)
For storing things other than git repos, GitHub is heavily invested in MySQL. AFAIK all of GitHub is hosted on their own hardware.
https://githubengineering.com/mysql-high-availability-at-git...
Re:Status.github.com: “We're failing over a data storage sys...
by rurban on Monday October 22, 2018 @03:44AM
(#18272866)
This blog post describes exactly the scenario we were experiencing here. A master (single writer) failure, with missing fail over. You can only guess what went wrong with this plan. Looks good on paper, but some unexpected network or HW or routing problem could have caused the problem to identify the single writer.
Re:Status.github.com: “We're failing over a data storage sys...
by reindeerer on Sunday October 21, 2018 @09:00PM
(#18271381)
I think it's obvious that it's their SQL storage that holds the website up that is pretty much in read-only mode, and has been for a couple hours now, not the git repos themselves.
Re:Status.github.com: “We're failing over a data storage sys...
by geerlingguy on Sunday October 21, 2018 @09:23PM
(#18271456)
Yeah, been trying to post comments and new issues and keep getting "405 Not Allowed" responses.
Re:Status.github.com: “We're failing over a data storage sys...
by friedman23 on Sunday October 21, 2018 @10:10PM
(#18271652)
I can push a branch but I cannot access the pull requests that I create off of that branch
[dead]
Re:Status.github.com: “We're failing over a data storage sys...
by jononor on Monday October 22, 2018 @02:51AM
(#18272666)
It is incredibly unlikely that there is any Azure in the production infrastructure. The acquisition is too recent for big changes like that.
Re:Status.github.com: “We're failing over a data storage sys...
by paavohtl on Monday October 22, 2018 @02:53AM
(#18272679)
It's actually so recent GitHub hasn't even been acquired yet. That's going to happen some time next year.
Re:Status.github.com: “We're failing over a data storage sys...
by dvfjsdhgfv on Monday October 22, 2018 @04:53AM
(#18273124)
I admire your confidence but will withhold my judgement until the postmortem is published.
In the enterprise space, a 'data storage system' could be an Array or a SAN or a lightpath etc, with usually quite long failover times. For an org like GitHub I'd think more like an object store (an Array-of-Hosts, if you will) or whatever storage mechanism holds their database files. Do they self host this sort of thing or is it an AWS/GCE/Azure service?
FWIW, all git commands are working fine for me (create a branch, push, colleagues can fetch my branch), but the UI doesn't show my branch & nor can I review/comment on PRs.