News for hackers. Stuff that matters.
Re:GitHub Is Down by twp on Wednesday August 14, 2024 @07:05PM (#41251425)
https://www.githubstatus.com/ reports no problems, but it's clearly down for a lot of people (including me).
Re:GitHub Is Down by tabbott on Wednesday August 14, 2024 @07:06PM (#41251440)
It is kinda amazing how consistently status pages show everything fine during a total outage. It's not that hard to connect a status page to end-to-end monitoring statistics...
Re:GitHub Is Down by blinded on Wednesday August 14, 2024 @07:10PM (#41251485)
From my experience this requires a few steps happen first:

- an incident be declared internally to github

- support / incident team submits a new status page entry (with details on service(s) impact(ed))

- incident is worked on internally

- incident fixed

- page updated

- retro posted

Even aws now seems to have some automation for their various services per region. But it doesn't automatically show issues because it could be at the customer level or subset of customers, or subset of customers if they are in region foo in AZ bar, on service version zed vs zed - 1. So they chose not to display issues for subsets.

I do agree it would be nice to have logins for the status page and then get detailed metrics based on customerid or userid. Someone start a company to compete with statuspage.

Re:GitHub Is Down by cortesoft on Wednesday August 14, 2024 @07:08PM (#41251460)
There is always going to be SOME delay between the outage and the status page, although 5 minutes is probably enough time where it should be updated
Re:GitHub Is Down by thund on Wednesday August 14, 2024 @07:12PM (#41251507)
after several minutes the status page is still showing all is fine.

For a service like GH, anything more than 30 secs is unacceptable

Re:GitHub Is Down by x86a on Wednesday August 14, 2024 @07:40PM (#41251829)
That is very unrealistic. Infrastructure monitoring at that scale won't even be collecting metrics at that interval.

And simple HTTP monitoring would be too flappy for a public status page.

Re:GitHub Is Down by aeonik on Thursday August 15, 2024 @07:38AM (#41254859)
What monitoring tools are you using? I know a ton that can do 30 seconds or less at scale. I'm fact, I'm pretty sure all the big players can do that.
Re:GitHub Is Down by frabjoused on Wednesday August 14, 2024 @07:07PM (#41251449)
It's simply too soon for the status page to report the anomaly, is my guess. It's been down for 4 minutes.
Re:GitHub Is Down by thih9 on Wednesday August 14, 2024 @07:12PM (#41251509)
4 minutes is a long time for something that could have been an automated check.

For the record, the status page eventually got updated - around 7 minutes after this submission was created.

Re:GitHub Is Down by owyn on Wednesday August 14, 2024 @09:33PM (#41252379)
Once in the past I did actually have an incident where the site went down so hard that the tool that we used to update the status page didn't work. We did move it to a totally external and independent service after that. The first service we used was more flaky than our actual site was, so it kept showing the site down when it wasn't. So then we moved to another one, etc. Job security. :)
Re:GitHub Is Down by beefsack on Thursday August 15, 2024 @02:08AM (#41253492)
They say you shouldn't host status pages on the same infrastructure that it is monitoring, but in a way that makes it much more accurate and responsive in outages!
Re:GitHub Is Down by kredd on Wednesday August 14, 2024 @07:07PM (#41251454)
It went down literally 3 minutes ago (I was in the middle of writing a PR comment), let's see if their cron job kicks in and reports the issue.
Re:GitHub Is Down by thund on Wednesday August 14, 2024 @07:13PM (#41251525)
it's starting to show now, about 10 minutes after the issue started
Re:GitHub Is Down by agosz on Wednesday August 14, 2024 @07:13PM (#41251526)
It's showing a few incidents now. Some things are still green though that don't seem to be working.