News for hackers. Stuff that matters.
Re:GitHub Is Down by fragmede on Wednesday August 14, 2024 @07:27PM (#41251690)
from where? they don't only have one load balancer, so you'd still have the problem of the page showing green when it's not loading for some folk?
Re:GitHub Is Down by ergocoder on Wednesday August 14, 2024 @07:40PM (#41251822)
At Github's scale, why wouldn't they put a ping monitor from every continent at least?

Then, you would show the status based on the continent.

Re:GitHub Is Down by intelVISA on Wednesday August 14, 2024 @11:11PM (#41252839)
That would be self-defeating given that it's a Rails app.
Re:GitHub Is Down by fragmede on Thursday August 15, 2024 @12:09AM (#41253024)
Where on the continent? GitHub is undoubtedly doing blackbox testing internally and has multiple such monitors but that's not going to capture every customer's route to them, leading to the same problem - customers experience GitHub being down, despite monitoring saying it's mostly up. Thus the impass. Even doing whitebox testing, where you know the internals and can this place sensors intelligently, even just for ingress, you're still at the mercy of the Internet.

If a sensor that's basically in the same datacenter says you're up, but the route into the datacenter is down, then what? multiply this by the complexity of the whole site, and monitoring it all with 100% fidelity is impossible. Not that it's not worth it to try, there's a team at GitHub that works on monitoring, but beyond motivation about keeping the SLA up, as a customer, unless you notice it's down, is it really down? In a globally distributed system, downtime, except for catastrophic downtime like this, is hard to define on a whole-site basis for all customers.

Re:GitHub Is Down by laserlight on Thursday August 15, 2024 @01:35AM (#41253370)
> 100% fidelity is impossible

I don't think anybody asked for 100% fidelity. We are talking about a complete outage that affected at least North America and Europe. If the status page shows green in such a case, its fidelity is around 50%. People expect better from GitHub.

Re:GitHub Is Down by fragmede on Thursday August 22, 2024 @01:59AM (#41317264)
The amount of moaning that the status page wasn't updated in 0 seconds and had the wrong status for entire minutes is what leads me to believe that no, users do expect 100a% fidelity.

Total outages are rare enough, and there's enough other work, that spending time building a system for that, just doesn't seem like the best use of their time. though I'm biased, having faced that exact question from the inside, at different company.

Re:GitHub Is Down by ergocoder on Thursday August 15, 2024 @03:34AM (#41253838)
> monitoring it all with 100% fidelity is impossible

This is impossible regardless of how godlike the design is... Nobody is asking for 100% fidelity.