News for hackers. Stuff that matters.
Re:Ask HN: Is GitHub down? by stefanos82 on Thursday June 29, 2023 @01:46PM (#36523974)
So down right now...I wonder why they still use https://www.githubstatus.com/ that reports everything is alright when it's not!
Re:Ask HN: Is GitHub down? by Shekelphile on Thursday June 29, 2023 @01:51PM (#36524056)
Pretty much every company has been shown to have fake status pages at this point.
Re:Ask HN: Is GitHub down? by AYBABTME on Thursday June 29, 2023 @02:06PM (#36524342)
Status pages are updated by humans and the humans need to (1) realize there's a problem and (2) understand the magnitude of the problem and (3) put that on the status page.

It's not fake, it's just a human process. And automating this would be error prone just the same.

Re:Ask HN: Is GitHub down? by wsatb on Thursday June 29, 2023 @02:19PM (#36524561)
I wouldn't necessarily call them fake, but the issue often has to be big enough for most companies to admit to it. AWS often has smaller outages that they will never acknowledge.
Re:Ask HN: Is GitHub down? by Macuyiko on Thursday June 29, 2023 @02:09PM (#36524403)
Very good points. Meanwhile I have clients asking me why they can't have a status page to which I reply: you can, but ultimately to be completely fail proof it will be a human updating it slowly. To which they reply: but GitHub or X does it...

Very infuriating, that.

Re:Ask HN: Is GitHub down? by AYBABTME on Thursday June 29, 2023 @02:15PM (#36524510)
There's some nice tooling these days for this. E.g. https://firehydrant.com/ and https://incident.io both make this a faster, more embedded process.
Re:Ask HN: Is GitHub down? by sjwhitworth on Thursday June 29, 2023 @02:46PM (#36524926)
Hey, incident.io CEO here! Thanks for mentioning us.
Re:Ask HN: Is GitHub down? by amandamacleod on Thursday June 29, 2023 @05:54PM (#36527286)
And Jeli.io for this! With the Statuspage integration, you can set the status, impact, write a message for customers, and select impacted components all without leaving Slack. Statuspage gets updated with a click of a button.
Re:Ask HN: Is GitHub down? by jachee on Thursday June 29, 2023 @04:35PM (#36526332)
Also (2b) convince their boss that the “optics” are better to update sooner than later.
Re:Ask HN: Is GitHub down? by ezekg on Thursday June 29, 2023 @01:58PM (#36524200)
Pretty much. They want the burden of proof for SLAs to fall on the customer, not on themselves. If a customer has to prove that an outage specifically affected them, they are much less likely to have a successful case against the failure to meet their SLA.

(Not directed at GitHub specifically, but at bogus status pages.)

Re:Ask HN: Is GitHub down? by wsatb on Thursday June 29, 2023 @02:03PM (#36524290)
From my experience, GitHub is the best out there when it comes to updating their status page.
Re:Ask HN: Is GitHub down? by Night_Thastus on Thursday June 29, 2023 @02:03PM (#36524293)
Really? Why?

That's so disappointing.

Re:Ask HN: Is GitHub down? by cududa on Thursday June 29, 2023 @02:15PM (#36524502)
Two technical reasons capstoned by driving business motivation:

-False positives -Short outages that last a minute or three

Ultimately, SLA's and uptime guarantees. That way, a business can't automatically tally every minute of publicly admitted downtime against the 99.99999% uptime guarantee, and the onus to prove a breach of contract is on the customer

Re:Ask HN: Is GitHub down? by klysm on Thursday June 29, 2023 @02:03PM (#36524275)
fake and not automated are pretty different
Re:Ask HN: Is GitHub down? by munk-a on Thursday June 29, 2023 @01:54PM (#36524112)
https://downdetector.com/status/github/ is a far more reliable source - it's just powered by user reports and often will show issues long before the status page ever receives an update.
Re:Ask HN: Is GitHub down? by jachee on Thursday June 29, 2023 @04:34PM (#36526318)
Keep in mind that downdetector can be brigaded and/or show knock-on problems instead of root causes. e.g. A couple weeks ago there were fairly major spikes across a rather huge variety of services on there, but it turned out that it was actually Comcast that was having trouble, rather than any of the “down” services.
Re:Ask HN: Is GitHub down? by SV_BubbleTime on Thursday June 29, 2023 @01:48PM (#36524004)
If it takes someone to manually change it from green to red, that does seem to defeat the purpose.
Re:Ask HN: Is GitHub down? by jabart on Thursday June 29, 2023 @01:52PM (#36524082)
No it doesn't. The amount of false alarm alerts you can get with internet based monitoring is more than 0. You could have a BGP route break things for one ISP your monitoring happens to use. You could have a failover event happening where it takes 30 seconds for everything to converge. I have multiple monitors on my app at 1 minute intervals from different vendors and ALWAYS a user will email us within 5 seconds of an issue. It's not realistic for a company to have automatic status updates trigger things without a person manually reviewing them because too many things can go wrong on the automatic status update to cause panic.
Re:Ask HN: Is GitHub down? by wongarsu on Thursday June 29, 2023 @01:59PM (#36524206)
Most paid status monitoring services cover BGP route problems and ISP issues by only flagging an event if it's detected from X geographically diverse endpoints.

For the 30 seconds where you wait for failover to complete: that is a 30 second outage. It's not necessarily profitable to admit to it, but showing it as a 30 second outage would be accurate

Re:Ask HN: Is GitHub down? by jabart on Thursday June 29, 2023 @02:36PM (#36524796)
TCP default is more than 30 seconds. The internet itself has about a 99.9% uptime. If one company showed every 30 second blip on their outage page all their competitors would have that screenshot on the first page of their pitch deck even if they also had the same issue. 2-5 minutes is reasonable for a public service to announce an outage.
Re:Ask HN: Is GitHub down? by jabart on Thursday June 29, 2023 @02:38PM (#36524822)
Forgot about that centurylink BGP infinite loop route bug they had where it took down their whole system nationwide. A lot of monitoring services showed red even though it was one ISP that was done.
Re:Ask HN: Is GitHub down? by lucb1e on Thursday June 29, 2023 @01:54PM (#36524125)
Who would panic? If nobody notices it's out because it's not, then nobody is going to be checking the status page. And if they do see the status page showing red while it's up, it's not like they're going to be unhappy about their SLA being met.

Maybe you want human confirmation on historic figures, but the live thing might as well be live.

Re:Ask HN: Is GitHub down? by AYBABTME on Thursday June 29, 2023 @02:09PM (#36524405)
Not really, things fail in unexpected ways. Automated anomaly detection is notoriously error prone, leading to a lot of false positive and false negatives, in the trivial case of monitoring a single timeseries. For a system the size of GitHub, you need to monitor a whole host of things and if it's quasi impossible to do one timeseries well, there's basically no hope of doing automated many timeseries anomaly detection with a signal-to-noise ratio that's better than "humans looking at the thing and realizing it's not going well".

There's stuff like this that can't be automated well. The automated result is far worse than the human-based alternative.

Re:Ask HN: Is GitHub down? by distortionfield on Thursday June 29, 2023 @01:53PM (#36524092)
Unknown unknowns means you can have catastrophic system failures that automated alerts don't detect.
Re:Ask HN: Is GitHub down? by numbsafari on Thursday June 29, 2023 @01:56PM (#36524159)
I bet they could teach Co-Pilot to create a PR to make the change, and build some GitHub actions to automatically merge those changes.
Re:Ask HN: Is GitHub down? by evulhotdog on Thursday June 29, 2023 @01:52PM (#36524072)
Yep, and when money comes into play when you're supposed to meet SLAs, you certainly don't want it being automatic.
Re:Ask HN: Is GitHub down? by klysm on Thursday June 29, 2023 @01:52PM (#36524080)
Possibly, but sometimes with failures this bad you can't get to the page to update it.
Re:Ask HN: Is GitHub down? by munk-a on Thursday June 29, 2023 @01:55PM (#36524140)
There was that hilarious multi-hour AWS failure a while back where the status page was updated via one of their internal services... and that service went down as part of the outage.
Re:Ask HN: Is GitHub down? by maschultz on Friday June 30, 2023 @02:53PM (#36540288)
Maybe the status page is down - it needs a status page to tell us if the status page is down