News for hackers. Stuff that matters.
Re:GitHub Is Down by sebmellen on Wednesday August 14, 2024 @07:06PM (#41251433)
I've never seen an outage this big. Even the homepage doesn't load. We've had recurrent issues with Actions not running, but this seems a lot bigger.

The status page says all is well, though: https://www.githubstatus.com/. Hilarious.

Re:GitHub Is Down by karmakaze on Wednesday August 14, 2024 @07:11PM (#41251500)
I get the angry unicorn page "No server is currently available to service your request. Sorry about that. Please try refreshing and contact us if the problem persists. Contact Support — GitHub Status — @githubstatus" with that last link going to https://x.com/githubstatus showing "GitHub Status Oct 22, 2018 Everything operating normally."
Re:GitHub Is Down by kalkin on Wednesday August 14, 2024 @07:18PM (#41251592)
I think this is because logged-out Twitter now shows top Tweets of all time from a user, rather than most recent Tweets.

Good reason why companies shouldn't be using Twitter/X for status updates anymore!

Re:GitHub Is Down by xp84 on Wednesday August 14, 2024 @07:37PM (#41251789)
Thank you! I was wondering why all I could see was useless content there!
Re:GitHub Is Down by gamer191 on Wednesday August 14, 2024 @07:42PM (#41251857)
Re:GitHub Is Down by TwiztidK on Wednesday August 14, 2024 @07:21PM (#41251622)
The era of Twitter/X status pages needs to come to an end given how unusable it is if you aren't logged in.
Re:GitHub Is Down by blt on Thursday August 15, 2024 @01:17AM (#41253299)
Making logins required to view twitter was the ultimate bed shitting move. The whole point of twitter was to be a broadcast medium. Tweets were viewable without following or logging in. There is a huge vacuum in that space now.
Re:GitHub Is Down by raxxorraxor on Thursday August 15, 2024 @03:18AM (#41253773)
For most (social media) platforms really. Management believes it would force users to sign up, but in reality the platform just becomes less relevant because of that limitation. Not even talking about search crawlers.

An all around stupid decision. That said, if management is that shitty, the platform probably won't be attractive for long anyway.

Facebook/Instagram were successful despite that to a degree, but this decision probably still did a lot of damage to their relevancy and user numbers.

Re:GitHub Is Down by disgruntledphd2 on Thursday August 15, 2024 @03:38AM (#41253853)
> Facebook/Instagram were successful despite that to a degree, but this decision probably still did a lot of damage to their relevancy and user numbers.

FB/IG/Whatsapp have half of humanity logging into their services once per month, so I'm not sure how much better they could be doing if they didn't have a login wall.

Meanwhile, Twitter (with no login wall) never broke 500mn. Like, personally I totally take your point about status updates but I'd have used my Twitter account a lot more if I'd needed to log in to see the content.

Re:GitHub Is Down by blt on Thursday August 15, 2024 @05:06PM (#41260358)
I don't really agree with that. Facebook was originally about mirroring your real-life social network. In the mid 00's nobody was trying to get likes from strangers on Facebook.

Instagram is closer to broadcast, but it was always closely tied to the mobile app experience and the "follower" mentality. People didn't really share links to Instagram posts in other online venues in the beginning.

Twitter was always unique. It existed before smartphones, and there was a good chunk of years where people without smartphones would read twitter posts on desktops. Its producer/consumer distribution is much more skewed, many twitter users never post. Tweets were always getting posted to places like HN, reddit, discussed in news articles, etc.

I think Twitter's (former) position as a broadcast medium à la TV, radio, and newspapers is unique among social networks. There's a reason why Twitter was the place for journalists, politicians, academics, fire departments, web service status alerts, etc.

Re:GitHub Is Down by Lanedo on Wednesday August 14, 2024 @07:32PM (#41251745)
Twitter now has:

We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.

https://x.com/githubstatus/status/1823864449494569023

Re:GitHub Is Down by Lanedo on Wednesday August 14, 2024 @07:46PM (#41251890)
Github seems to be coming back up:

https://downdetector.com/status/github/

Re:GitHub Is Down by temp0826 on Wednesday August 14, 2024 @08:03PM (#41251999)
Used to work ops at AWS. I don't know if it's still the case but it required VERY HIGH management approval to actually flip any lights on their "status page" (likely it was referenced in some way for SLAs and refunding customers).
Re:GitHub Is Down by smsm42 on Wednesday August 14, 2024 @08:14PM (#41252047)
[delayed]
Re:GitHub Is Down by purkka on Wednesday August 14, 2024 @07:16PM (#41251579)
I have to wonder how a company at the scale of GitHub can be so bad at keeping track of their status.

Now 4 out of 10 services are marked as "Incident", yet most of the others are also completely dead.

Re:GitHub Is Down by x86a on Wednesday August 14, 2024 @07:20PM (#41251610)
This is intentional. It's mostly a matter of discussing how to communicate it publicly and when to flip the switch to start the SLA timer. Also coordinating incident response during a huge outage is always challenging.
Re:GitHub Is Down by thiagocsf on Wednesday August 14, 2024 @10:30PM (#41252643)
That it may be but there’s no excuse.

Declare an incident first, investigate later.

Cheating SLAs by delaying the incident is a good way to erode trust within and without.

Re:GitHub Is Down by antimemetics on Thursday August 15, 2024 @12:56AM (#41253207)
> Declare an incident first, investigate later.

If that would be the best way to deal with it- why is literally no one doing it this way and what does that tell you?

Re:GitHub Is Down by ErikBjare on Thursday August 15, 2024 @04:16AM (#41254010)
[delayed]
Re:GitHub Is Down by adgjlsfhk1 on Thursday August 15, 2024 @01:14AM (#41253285)
because it involves admitting that you messed up which companies are often disensentivized to do
Re:GitHub Is Down by xuancanh on Wednesday August 14, 2024 @07:29PM (#41251706)
It's because of the way most companies build their status dashboards. There are usually at least 2 dashboards, one internal dashboard and one external dashboard. The internal dashboard is the actual monitoring dashboard, where it will be hooked up with other monitoring data sources. The external status dashboard is just for customer communication. Only after the outage/degradation is confirmed internally, then the external dashboard will be updated to avoid flaky monitors and alerts. It will also affect SLAs so it needs multiple levels of approval to change the status, that's why there are some delays.
Re:GitHub Is Down by ParetoOptimal on Wednesday August 14, 2024 @07:43PM (#41251859)
> The external status dashboard is just for customer communication. Only after the outage/degradation is confirmed internally, then the external dashboard will be updated to avoid flaky monitors and alerts. It will also affect SLAs so it needs multiple levels of approval to change the status, that's why there are some delays.

This defeats the purpose of a status dashboard and is effectively useless in practice most of the time from a consumers point of view.

Re:GitHub Is Down by consteval on Thursday August 15, 2024 @09:40AM (#41255939)
From a business perspective, I think given the choice to lie a little bit or be brutally honest with your customers, lying a bit is almost always the correct choice.
Re:GitHub Is Down by ParetoOptimal on Thursday August 15, 2024 @12:18PM (#41257687)
My ideal would be if regulations which made it necessary that downtime metrics had to be reported with at most somewhere between a 10m and 30m delay as "suspected reliability issue".

If your reliability metrics have lots of false positives, that's on you and you'll have to write down some reason why those false positives exist every time.

Then that company could decide for itself whether to update manually with "not a reliability issue because X".

This lets consumers avoid being gaslighted and businesses don't technically have to call it downtime.

Re:GitHub Is Down by insane_dreamer on Thursday August 15, 2024 @10:03AM (#41256196)
Liability is their primary concern
Re:GitHub Is Down by kinduff on Wednesday August 14, 2024 @07:12PM (#41251521)
They are flipping the switches now, status page just changed.
Re:GitHub Is Down by ergocoder on Wednesday August 14, 2024 @07:09PM (#41251481)
I wonder why the status just doesn't ping github.com for 200. That seems easy to do.
Re:GitHub Is Down by tinyhitman on Wednesday August 14, 2024 @07:10PM (#41251487)
delaying SLA
Re:GitHub Is Down by sebmellen on Wednesday August 14, 2024 @07:10PM (#41251491)
This is at least a multi-million dollar payout (if they admit to it).

All GitHub Pages say

> We're having a really bad day.

> The Unicorns have taken over. We're doing our best to get them under control and get GitHub back up and running.

Re:GitHub Is Down by ljahier on Wednesday August 14, 2024 @07:58PM (#41251970)
At the moment, all github services seem to be restored, and the github status indicates that the problem is still ongoing. I don't think it's related to the SLA, but rather to the monitoring, which is not live. There are a few minutes of delay.
Re:GitHub Is Down by cbates on Wednesday August 14, 2024 @07:24PM (#41251658)
Seems slightly unproffesional for a massive company like Github/Microsoft.
Re:GitHub Is Down by colimbarna on Wednesday August 14, 2024 @07:51PM (#41251929)
It's closer to the truth than you usually get. They're having a bad day, it's completely true. It's the start of my day, but I guess this is the middle of the night for them. There's no such thing as unicorns, but that just highlights the metaphorical nature of the remaining claim - getting Unicorns under control means solving their problems. Normally "professional" corporate speak means avoiding saying anything whose meaning is plain on its face and disconfirmable while avoiding the implication that the company is run and operated by humans. This is a model. (Obviously the came up with the message in advance, which just goes to show that someone in the company is well enough rounded to know that if it is displayed, they're having a bad day.)
Re:GitHub Is Down by wrs on Wednesday August 14, 2024 @09:11PM (#41252277)
GitHub is (was?) a Rails application, so it was probably originally running behind Unicorn [0], if it isn’t still. So the unicorns are (were) real.

[0] https://en.wikipedia.org/wiki/Unicorn_(web_server)

Re:GitHub Is Down by xp84 on Wednesday August 14, 2024 @07:38PM (#41251808)
I disagree. This hurts no one, and not everything needs to be sanitized and painted over with bland corporatespeak.
Re:GitHub Is Down by majewsky on Thursday August 15, 2024 @04:57AM (#41254194)
I don't think they were asking for corporate speak. But at least I would find a plain technical error message like "cannot contact file server" much more respectable than something like "unicorns are hugging our servers uwu".
Re:GitHub Is Down by COMMENT___ on Thursday August 15, 2024 @02:30PM (#41259013)
This “ironic” and “humorous” style of errors and UI captions is the actual new corporate speak. I’d prefer dumb error messages rather than some shit someone over the ocean thinks is smart and humorous. And it’s not funny at all when it’s a global outage impacting my business and my $$$.
Re:GitHub Is Down by fragmede on Wednesday August 14, 2024 @07:27PM (#41251690)
from where? they don't only have one load balancer, so you'd still have the problem of the page showing green when it's not loading for some folk?
Re:GitHub Is Down by ergocoder on Wednesday August 14, 2024 @07:40PM (#41251822)
At Github's scale, why wouldn't they put a ping monitor from every continent at least?

Then, you would show the status based on the continent.

Re:GitHub Is Down by intelVISA on Wednesday August 14, 2024 @11:11PM (#41252839)
That would be self-defeating given that it's a Rails app.
Re:GitHub Is Down by fragmede on Thursday August 15, 2024 @12:09AM (#41253024)
Where on the continent? GitHub is undoubtedly doing blackbox testing internally and has multiple such monitors but that's not going to capture every customer's route to them, leading to the same problem - customers experience GitHub being down, despite monitoring saying it's mostly up. Thus the impass. Even doing whitebox testing, where you know the internals and can this place sensors intelligently, even just for ingress, you're still at the mercy of the Internet.

If a sensor that's basically in the same datacenter says you're up, but the route into the datacenter is down, then what? multiply this by the complexity of the whole site, and monitoring it all with 100% fidelity is impossible. Not that it's not worth it to try, there's a team at GitHub that works on monitoring, but beyond motivation about keeping the SLA up, as a customer, unless you notice it's down, is it really down? In a globally distributed system, downtime, except for catastrophic downtime like this, is hard to define on a whole-site basis for all customers.

Re:GitHub Is Down by laserlight on Thursday August 15, 2024 @01:35AM (#41253370)
> 100% fidelity is impossible

I don't think anybody asked for 100% fidelity. We are talking about a complete outage that affected at least North America and Europe. If the status page shows green in such a case, its fidelity is around 50%. People expect better from GitHub.

Re:GitHub Is Down by fragmede on Thursday August 22, 2024 @01:59AM (#41317264)
The amount of moaning that the status page wasn't updated in 0 seconds and had the wrong status for entire minutes is what leads me to believe that no, users do expect 100a% fidelity.

Total outages are rare enough, and there's enough other work, that spending time building a system for that, just doesn't seem like the best use of their time. though I'm biased, having faced that exact question from the inside, at different company.

Re:GitHub Is Down by ergocoder on Thursday August 15, 2024 @03:34AM (#41253838)
> monitoring it all with 100% fidelity is impossible

This is impossible regardless of how godlike the design is... Nobody is asking for 100% fidelity.

Re:GitHub Is Down by bigiain on Wednesday August 14, 2024 @08:59PM (#41252235)
To be fair - I really couldn't care less is the homepage is loading or not.

So long as I can fetch/commit to my repos, pretty much everything else is of secondary, tertiary, or no real importance to me.

(At work, I do indeed have systems running that monitor 200 statuses from client project homepages, almost all of which show better that 99.999% uptimes. And are practically useless. Most of them also monitor "canary" API requests which I strive to keep at 99.99% but don't always manage to achieve 99.9% - which is the very best and most expensive SLA we'll commit to.)

Re:GitHub Is Down by RIMR on Wednesday August 14, 2024 @07:13PM (#41251535)
Wow, the status page only just now started reporting issues, and it still doesn't seem to communicate the scale of the issue.

People use this page for guidance. I guess now we know how much it can be trusted.

Re:GitHub Is Down by ikiris on Wednesday August 14, 2024 @07:15PM (#41251563)
It’s used to ease their comms, not a real time status board pointing at their monitoring.
Re:GitHub Is Down by manquer on Wednesday August 14, 2024 @07:13PM (#41251533)
Status page updates with "degraded availability". lol
Re:GitHub Is Down by rvz on Wednesday August 14, 2024 @07:27PM (#41251691)
Looks like we have a full house outage at GitHub with everything down. Much worse than the so-called Twitter / X recent speed-bump that was screeched at and quickly forgotten.

I don't think GitHub has recovered from the monthly incidents that keeps occurring. Quite frankly it is the expectation that something will go down every month at GitHub which shows how unreliable the service is and this has happened for years.

I guess this 4 year old prediction post really aged well after all about self-hosting and not going all in on Github [0]

[0] https://news.ycombinator.com/item?id=22868406

Re:GitHub Is Down by dataspun on Wednesday August 14, 2024 @07:40PM (#41251832)
statute of limitations for HN comment predictions is 3 years.
Re:GitHub Is Down by saul-paterson on Thursday August 15, 2024 @02:33AM (#41253583)
FWIW, our self-hosted Gitea instance has not had a single second of unplanned downtime in five years we've been running it. And there wasn't much _planned_ downtime because it's really easy to upgrade (pull a new image and recreate the container — takes out the instance for maybe 15 seconds late at night), and full backups are handled live thanks to zfs.

Migration to a new host takes another 15 seconds thanks to both zfs and containers.

I don't know how many GitHub downtime reports I've seen during that time, we're probably into high dozens by now.

Re:GitHub Is Down by chrisallenlane on Thursday August 15, 2024 @08:07AM (#41254993)
I've been running Gitea on my homelab for a few months now. It's fantastic. It's like a snapshot of a point in time when GitHub was actually good, before it got enshittified by all of the social and AI nonsense.

I've been moving most of my projects off of GitHub and into Gitea, and will continue to do so.

Re:GitHub Is Down by TacticalCoder on Thursday August 15, 2024 @08:11AM (#41255014)
> I've never seen an outage this big.

I remember a time when systems would boast about their "five nines" uptime. It was before anything "cloud" appeared.

Re:GitHub Is Down by rozenmd on Thursday August 15, 2024 @05:23PM (#41260520)
Here, we caught 35 minutes of downtime: https://github.onlineornot.com/incidents/6Yyj8YWD94zE