News for hackers. Stuff that matters.
Re:GitHub Outage Currently Ongoing – Issues, PRs, Notifications by nickpsecurity on Thursday August 22, 2019 @08:13AM (#20767158)
The problem is bad design and/or integration of components. VMS clusters over leased lines went years without outages. Record was 17 years at a rail yard. They also did rolling upgrades across both different OS versions and CPU architectures. The methods are public if any competitors want to match or exceed them:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na...

These modern cloud services using Linux ecosystem are doing rolling outages instead. I'm amazed people keep making them dependencies. Maybe their local systems using similar technology went down more often. Maybe this is an improvement. I'd still put mine on an OpenBSD or OpenVMS cluster if it was business-critical with lots of money on the line. I want it staying up, up, up. :)

Re:GitHub Outage Currently Ongoing – Issues, PRs, Notifications by mugsie on Thursday August 22, 2019 @08:31AM (#20767288)
Modern apps are designed to deal with part of them being down - it is the core upgrade strategy.

VMS clusters were (and modern equivalents are) awesome, but they have a different speed of innovation / development.

It is a trade off - perfect uptime, but that new feature could take a year, or 99.9% uptime, but it can be written and deployed tomorrow.

Re:GitHub Outage Currently Ongoing – Issues, PRs, Notifications by nickpsecurity on Thursday August 22, 2019 @09:38AM (#20767884)
Their development velocity might have been due to their management or internal culture. You can get most of those results faster and on a budget. The best evidence was the startup behind FoundationDB continually improving their product in highly-competitive space while doing QA like this:

https://www.youtube.com/watch?v=4fFDFbi3toc

Generally, on top of code reviews, one can get high return with minimal labor and hardware with a combo of Design-by-Contract, contract/spec/property-based testing, low-F.P. static analysis, and fuzzing with contracts in as runtime checks. That's my default recommendation.

In Github's case, they also might have access to both closed and open tools that Microsoft Research makes. MSR is awesome if you've never checked them out. Two examples applicable to system reliability and security:

https://www.microsoft.com/en-us/research/project/ironclad/

https://www.microsoft.com/en-us/research/publication/p-safe-...

Plus some of their other tools in various states of usability:

https://rise4fun.com/

Re:GitHub Outage Currently Ongoing – Issues, PRs, Notifications by fortran77 on Thursday August 22, 2019 @12:48PM (#20769987)
Our Erlang nodes are never down. We do live code updates and have gone several years now with no downtime.