News for hackers. Stuff that matters.
Re:Incident with Github.com by ieie3366 on Monday August 17, 2026 @09:57AM (#49330993)
To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside.

If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health

Re:Incident with Github.com by askonomm on Monday August 17, 2026 @10:01AM (#49331087)
That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.
Re:Incident with Github.com by datsci_est_2015 on Monday August 17, 2026 @10:16AM (#49331388)
It is interesting, how much money is being lost during this outage?

My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced out by end-of-year due to incompetence. The new executive has taken to firing employees much more eagerly than their predecessor, like some sort of Jack Welch acolyte.

Their firing has put a lot of things into perspective for me. Mostly, fuck "at-will" employment and its negative effect on the American social contract.

But also this "angry ranting" online that the original poster was referencing. Not everyone has the privilege to calmly respond to things that directly impact their livelihood.

Re:Incident with Github.com by theappsecguy on Monday August 17, 2026 @11:19AM (#49332489)
Any engineered system where an individual can accidentally cause a 7 figure outage is poorly designed. And engineering leadership that decider to terminate an individual due to such failure (as long as there was no malicious actions) is completely clueless.
Re:Incident with Github.com by starkshift on Monday August 17, 2026 @11:29AM (#49332651)
Captain obvious over here
Re:Incident with Github.com by rkapsoro on Monday August 17, 2026 @01:54PM (#49334979)
A story as old as time.
Re:Incident with Github.com by pkilgore on Monday August 17, 2026 @01:55PM (#49334992)
Name names.
Re:Incident with Github.com by datsci_est_2015 on Monday August 17, 2026 @02:01PM (#49335110)
Local (to us) firm. Two main reasons I didn't:

  - Have never seen it mentioned on this forum, in any context.
  - We're potentially pursuing legal action.
Re:Incident with Github.com by red-iron-pine on Monday August 17, 2026 @04:03PM (#49336763)
names or it didn't happen, mate

50% of social media traffic is bots and i want to believe

Re:Incident with Github.com by datsci_est_2015 on Tuesday August 18, 2026 @03:25PM (#49351239)
I'll keep you guys in mind if the lawsuit is successful and we're allowed to publicly speak about it. I believe it's a pretty compelling (and sympathetic) story that deserves to be told. But it's still far too soon for me to be shouting the name from the rooftops.

My SO was fired - this isn't a Google review where we were treated poorly at a restaurant.

Re:Incident with Github.com by red-iron-pine on Wednesday August 19, 2026 @09:01AM (#49361064)
that's fair, and ultimately who gives af about online randos, but at the same time skepticism is required
Re:Incident with Github.com by TwelveEyes on Monday August 17, 2026 @02:35PM (#49335632)
Is responding calmly a privilege to have, or a kindness to yourself?
Re:Incident with Github.com by mplewis on Monday August 17, 2026 @02:41PM (#49335705)
don't do this
Re:Incident with Github.com by itemize123 on Tuesday August 18, 2026 @10:34PM (#49355908)
how is it mutually exclusive
Re:Incident with Github.com by Waterluvian on Monday August 17, 2026 @10:09AM (#49331248)
I'd be curious what the statistics might actually be for people who are directly affected because their business is suffering vs. people affected because their employer's business is suffering.
Re:Incident with Github.com by 6LLvveMx2koXfwn on Monday August 17, 2026 @10:20AM (#49331456)
Most people's employment prospects are directly correlated with their employers ability to make money.
Re:Incident with Github.com by john_strinlai on Monday August 17, 2026 @10:30AM (#49331640)
most people's employers arent firing people over a few hours of github outage.

not to mention that any business which could potentially lose enough money that they would need to let go of developers from a github outage should probably already have some business continuity plans in place.

Re:Incident with Github.com by polycaster on Monday August 17, 2026 @10:17AM (#49331403)
For Germany it's roughly 1:11 if you go by the self-employment rate (~8% of the workforce).
Re:Incident with Github.com by nullify88 on Monday August 17, 2026 @11:34AM (#49332725)
GitHub Enterprise Cloud has been chugging along with no issues. I hope your critical infrastructure isn't dependent on a free tier / service. And that you have a business continuity process in place.
Re:Incident with Github.com by 946789987649 on Monday August 17, 2026 @11:49AM (#49332975)
Are you sure? We have enterprise and I couldn't even access our repo briefly.
Re:Incident with Github.com by nullify88 on Monday August 17, 2026 @12:02PM (#49333177)
We have data residency in the EU and didn't notice anything.
Re:Incident with Github.com by roastedfunction on Monday August 17, 2026 @01:11PM (#49334360)
GHE.com has only been around since November 2024. Most GHEC customers I imagine are still stuck on the shared global infrastructure on github.com.
Re:Incident with Github.com by antaviana on Monday August 17, 2026 @10:09AM (#49331252)
"It's just money. It's made up. Pieces of paper with pictures on it so we don't have to kill each other just to get something to eat". Jeremy Irons in movie Margin Call
Re:Incident with Github.com by faeyanpiraat on Monday August 17, 2026 @10:24AM (#49331528)
You cant eat a quote
Re:Incident with Github.com by red-iron-pine on Monday August 17, 2026 @04:03PM (#49336771)
but you can download a car
Re:Incident with Github.com by parthdesai on Monday August 17, 2026 @10:32AM (#49331672)
How is github being down losing you guys money?
Re:Incident with Github.com by dewey on Monday August 17, 2026 @10:38AM (#49331775)
If you pay developers x money / day and one of their core tools is down for n hours during the day and they spend their money on HN instead that's pretty straight forward to calculate.
Re:Incident with Github.com by parthdesai on Monday August 17, 2026 @10:49AM (#49331987)
I meant more for their customers specifically
Re:Incident with Github.com by red-iron-pine on Monday August 17, 2026 @04:07PM (#49336824)
we push customer code on mondays. it is monday. code is not being pushed.

we do not bill.

the outages they promised their customers are now different.

this has costs for everyone downstream.

Re:Incident with Github.com by aurareturn on Monday August 17, 2026 @10:41AM (#49331843)
I was in the middle of a hot fix. Our pipeline goes through Github.
Re:Incident with Github.com by parthdesai on Monday August 17, 2026 @10:50AM (#49332002)
That's understandable, but surely there's a way to bypass it? You need to have a breakglass procedures lol
Re:Incident with Github.com by penultimatename on Monday August 17, 2026 @11:12AM (#49332375)
“lol just use a workaround” doesn’t work in an environment with hundreds or thousands of employees coupled with audit, security, and other legal requirements to ship software.
Re:Incident with Github.com by parthdesai on Monday August 17, 2026 @11:42AM (#49332839)
If you're actually bleeding money, you better believe you'll get permissions for a workaround, if you know what you're doing.

It all ties back to the OP, where the issue you've might not be as bad as you think. I have been in situations where we have dropped all procedures to push a hot fix because we were actively bleeding money, and in situations where you know there is an issue, and you let it be.

Re:Incident with Github.com by penultimatename on Monday August 17, 2026 @01:13PM (#49334381)
What an odd hill to die on. Maybe I just lost a $30M client because they wanted an issue fixed now. Maybe my company would get sued for violating their contractual obligations if someone went cowboy to bypass procedures.

Maybe I pay for a service and I want that service to work consistently during core business hours.

Re:Incident with Github.com by bmitc on Monday August 17, 2026 @01:21PM (#49334483)
"Just recreate GitHub internally."
Re:Incident with Github.com by red-iron-pine on Monday August 17, 2026 @04:07PM (#49336835)
"just redo the whole CI/CD pipeline bro, it's easy"
Re:Incident with Github.com by phoronixrly on Monday August 17, 2026 @10:48AM (#49331985)
Imagine Github being critical infrastructure for you... The ineptitude...
Re:Incident with Github.com by parthdesai on Monday August 17, 2026 @10:50AM (#49332018)
We use github quite a bit as well. Github being down currently is not actively losing us money, i.e. having customer impact.
Re:Incident with Github.com by kakacik on Monday August 17, 2026 @10:46AM (#49331941)
If its so critical why relying on it, and not having ie some mirror or some other way to handle any sort of outage like this. its not like Microsoft is your friend or good business partner, ever.

With every single of these enterprise 'cloud' offerings you are giving (almost) complete power over your business/project to somebody else who couldn't care less about your success or failure, you are simply irrelevant for them. I see it at work too, every time critical external systems go down whole bank stops still, just because few bucks were saved yearly on some on-prem servers.

Look at it this way, you are learning some important lesson today and finding great area of improvement for resiliency from now on.

Re:Incident with Github.com by AlexandrB on Monday August 17, 2026 @10:04AM (#49331160)
Boomer opinion but trusting third parties to be critical infrastructure, especially with no SLA in sight, will always end in tears. "The cloud" is very convenient, but its providers will never care about your infrastructure or your customers as much as you will.
Re:Incident with Github.com by Maxion on Monday August 17, 2026 @10:08AM (#49331225)
Older millennial here and I agree. If you rely on Github you should at least be have your processes such that you can work around it.
Re:Incident with Github.com by whynotmaybe on Monday August 17, 2026 @10:30AM (#49331629)
That's the reason why my build agent is self-hosted
Re:Incident with Github.com by otterley on Monday August 17, 2026 @10:22AM (#49331486)
3P-maintained infrastructure is what makes civilizations work efficiently. We're not all digging our own wells, generating our own electricity, and burning or burying our own garbage.
Re:Incident with Github.com by kalaksi on Monday August 17, 2026 @10:40AM (#49331829)
You are still supposed to be prepared for outages.
Re:Incident with Github.com by otterley on Monday August 17, 2026 @11:22AM (#49332541)
Agreed. So is the issue really then that people were inadequately prepared with backup plans and now they're suffering the consequences?

It's not all that different from, say, an AWS region having a service impact. People would rather complain about AWS than prepare and utilize a well-tested recovery plan to shift to a standby region. Oftentimes there's no fallback plan because the business already considered it and decided it was too costly relative to the benefit, but when the incident happens, they still can't help but complain. Humans being humans.

Re:Incident with Github.com by aenis on Monday August 17, 2026 @10:16AM (#49331385)
Boomer here as well, but I'd add that trusting your own org for critical infra usually also ends in tears. Most everything in IT involves failure, including in well designed systems designed by great engineers.

I worked for a few years in an exceedingly well capitalised place which ran everything in their own data centers, money no object, with a truck parked somewhere, ready to go, with a smaller version of our critical infra. We had a serious business-stopping outage once every 18 months or so, every time for fringe reasons one only learns about when trying to run a large data center. Its convenient to blame the cloud and pretend that self-hosting in private sector was so, so great with six nines.

Re:Incident with Github.com by 6031769 on Monday August 17, 2026 @10:41AM (#49331836)
At this point github is barely managing one nine.
Re:Incident with Github.com by aenis on Monday August 17, 2026 @10:50AM (#49332010)
True. I'd be very annoyed if I had anything mission critical on github. While we have everything in the cloud, we do actually self-host that part.
Re:Incident with Github.com by swills on Monday August 17, 2026 @11:11AM (#49332350)
Agree. But also, it's affecting everyone equally, whether they have a free personal account or are part of an enterprise account with SLA. Understanding the practical value of an SLA is an interesting problem.
Re:Incident with Github.com by raffael_de on Monday August 17, 2026 @10:08AM (#49331236)
+1 (as a millenial) ... especially given that setting up a git server for non-OSS company code isn't too much of a challenge really. also, no need to self-denigrate this reasonable opinion in preemptive obedience.
Re:Incident with Github.com by otterley on Monday August 17, 2026 @11:24AM (#49332563)
GitHub is way more than just a git repo host. It manages code reviews, merge (pull) requests, and has an entire CI/CD workflow engine in it. Replicating all that is a challenge that most orgs are not up to.
Re:Incident with Github.com by poisonborz on Monday August 17, 2026 @10:13AM (#49331330)
How did this become a boomer opinion? It is proved truth thousand times a day. Not that you shouldn't use third parties - but in this industry you can shrink this exposure to the minimum, and have plan B for anything else.
Re:Incident with Github.com by ethagnawl on Monday August 17, 2026 @10:35AM (#49331721)
You're right.

However, I'm also of the boomer opinion that you should get what you pay for. "Ranting online" about a service (you pay for) being unavailable is a reasonable reaction. It's not like they have a call center you can dial into for support ...

Re:Incident with Github.com by SideburnsOfDoom on Monday August 17, 2026 @10:11AM (#49331293)
People are already saying things like "We need a plan B in case we urgently need to deploy a fix to production, and GitHub Actions is unavailable again".

But in general, it's not feasible to do everything in house.

And I don't think that GitHub is devoid of SLA: https://github.com/customer-terms/github-online-services-sla

The issue is that they're not achieving two nines uptime in practice.

Re:Incident with Github.com by TZubiri on Monday August 17, 2026 @10:13AM (#49331326)
Boomer who has never had a business? Depending on third party suppliers is business as usual.
Re:Incident with Github.com by pigeons on Monday August 17, 2026 @11:02AM (#49332198)
Well it's not business at all today!
Re:Incident with Github.com by TZubiri on Monday August 17, 2026 @01:56PM (#49335031)
Arguably, git is decentralized, so contingency plans, like working with local and server repos, should still work.
Re:Incident with Github.com by BryanBeshore on Monday August 17, 2026 @10:06AM (#49331185)
Truly a boomer opinion. Respect!
Re:Incident with Github.com by advisedwang on Monday August 17, 2026 @03:06PM (#49336047)
Eeh, you always rely on someone else's infrastructure. Even if you are off cloud entirely and own your own datacenter, you still need peering/transit. You still need power (or at least fuel). You can't avoid depending on a DNS infrastructure.

Github does have an SLA: https://github.com/customer-terms/github-online-services-sla. But like virtually all SLAs, it's really just a token gesture. I have never seen an SLA that pays the losses you suffer due to the outage.

Re:Incident with Github.com by TZubiri on Monday August 17, 2026 @10:12AM (#49331309)
And it's not just the user's fault, GH spent a considerable effort in marketing to position themselves as such, see Github Actions and similar junk.
Re:Incident with Github.com by vaylian on Monday August 17, 2026 @02:13PM (#49335258)
> For a lot of us GitHub is critical infrastructure

Please read https://berthub.eu/articles/posts/cyber-security-pre-war-rea...

Re:Incident with Github.com by peterzat on Tuesday August 18, 2026 @09:14PM (#49355305)
A lot of casual tech folks I know don't understand GitHub's role in CI/CD, and think it's "just" revision control storage. It's natural, since many people have no reason to know about systems like ghcr.io and npm registry if they're storing vibe-coded personal projects.
Re:Incident with Github.com by jjice on Monday August 17, 2026 @10:03AM (#49331144)
> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory).

I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain.

You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org.

[0] https://damrnelson.github.io/github-historical-uptime/

Re:Incident with Github.com by twistedpair on Monday August 17, 2026 @10:16AM (#49331389)
They _are_ the LLM coding agent vendor, and are _owned_ by MS, the ~majority~ biggest shareholder of OpenAI. How can you NOT consider that 20x+ scaling in your capacity roadmap projections, where you are trying to get everyone to use these agents as part of your core OKRs?

Or, maybe your #1 IT priority was moving everything to Azure instead ;)

Re:Incident with Github.com by rafram on Monday August 17, 2026 @10:45AM (#49331914)
> the majority owner of OpenAI

Largest shareholder (27%), not majority.

Re:Incident with Github.com by chilipepperhott on Monday August 17, 2026 @11:33AM (#49332715)
Plurality should have been the operative word.
Re:Incident with Github.com by debarshri on Monday August 17, 2026 @10:09AM (#49331253)
It looks like right after Microsoft acquisition this started going off.

This is the outcome of violating single responsibility principle in business.

Re:Incident with Github.com by Topfi on Monday August 17, 2026 @10:08AM (#49331226)
Github being under the CoreAI division probably also doesn't help the engineers prioritize addressing infrastructure issues and makes using LLM load as an excuse feel self-inflicted. Akin to feeling sorry when a pyromaniacs house burns down...
Re:Incident with Github.com by chrisjj on Monday August 17, 2026 @10:30AM (#49331628)
They've had a couple of months and still have not brought in sufficient extra capacity?? What's their excuse?
Re:Incident with Github.com by eddieroger on Monday August 17, 2026 @10:00AM (#49331069)
For the amount of money enterprises are paying to GitHub, there is a reasonable (and contractual) expectation of uptime. I don't think my boss would find it to fun if I bailed work to hit the gym just because GitHub was down.
Re:Incident with Github.com by iamcoder18 on Monday August 17, 2026 @10:11AM (#49331281)
Yes, but GitHub enterprise has different uptime (https://us.githubstatus.com/posts/dashboard) and actually does have an SLA.

Edit: it seems GHE is also affected by the outage and isn't much better.

Re:Incident with Github.com by cyansmoker on Monday August 17, 2026 @10:36AM (#49331752)
As an enterprise user I can tell you that:

- we are getting no benefit beyond getting excuses replies to our emails - right or wrong, but we depend heavily on GitHub actions so this has a very real impact on us

Re:Incident with Github.com by twistedpair on Monday August 17, 2026 @10:29AM (#49331609)
Since upgrading to Cloud Enterprise... why didn't my GH uptime get any better? Guess on-prem GHE server is the only way to do that.
Re:Incident with Github.com by joeyhage on Monday August 17, 2026 @10:47AM (#49331949)
My understanding is the us.githubstatus.com is only if you are using GitHub enterprise with a custom subdomain and data residency in the US because then *some* of the infrastructure is separate from GitHub.com.
Re:Incident with Github.com by eddieroger on Monday August 17, 2026 @10:59AM (#49332137)
Yeah, I know, and it's down.
Re:Incident with Github.com by tedivm on Monday August 17, 2026 @10:51AM (#49332030)
This, like all of their dashboards, is complete bullshit. As a GitHub enterprise user i am absolutely affected by the outage today, and have been affected by all of the GitHub Cloud outages.

There is no separate enterprise platform unless you buy their self hosted server product.

Re:Incident with Github.com by leishman on Monday August 17, 2026 @10:02AM (#49331102)
> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

Which means they have a pricing problem. Rate limit non-paying users.

Re:Incident with Github.com by xnorswap on Monday August 17, 2026 @10:29AM (#49331606)
Just take away free Actions and it would surely solve a lot of their scale issues.

It's absurd that I have dozens of repos, many with GH Actions that run CI, test and then package and push to prod/package managers, and haven't paid GH anything.

Re:Incident with Github.com by anoplus on Monday August 17, 2026 @10:08AM (#49331223)
Right. It is not an excuse and they were notorious for outages before LLM era. Also, LLM can serve Github engineers as well, so we are all playing in the same field.
Re:Incident with Github.com by cluckindan on Monday August 17, 2026 @10:08AM (#49331227)
They already do
Re:Incident with Github.com by rzzzt on Monday August 17, 2026 @10:38AM (#49331781)
I can't check the commit history of some OSS projects without hitting a rate limit if I'm not signed in. My request rate is one request per (arbitrary time interval) at that point.
Re:Incident with Github.com by chrisjj on Monday August 17, 2026 @10:32AM (#49331669)
No. It means they have a capacity problem.
Re:Incident with Github.com by denysvitali on Monday August 17, 2026 @10:06AM (#49331200)
Re:Incident with Github.com by sausagefeet on Monday August 17, 2026 @10:03AM (#49331137)
Actually, I do have an important fix for a deal we are trying to close that does need to go out. And I am paying GitHub to host this. Once or twice, I can see, but GitHub's SLA is getting worse than just hosting it myself, and that's the entire reason I pay GitHub.
Re:Incident with Github.com by world2vec on Monday August 17, 2026 @10:03AM (#49331138)
Geez, sorry that the paid service my company paid for is down and I can't do my work.
Re:Incident with Github.com by sarjann on Monday August 17, 2026 @10:05AM (#49331166)
This has to be rage bait, this is a critical piece of infrastructure for many people. Stuff going down can lead to deployments failing and as you mentioned in many cases emergency hotfix's.

The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.

Re:Incident with Github.com by john_strinlai on Monday August 17, 2026 @10:07AM (#49331221)
>The idea that we should be fine with this unreliability is just amazing.

the comment doesnt say you should be "fine" with the unreliability.

they are saying people shouldnt get so emotionally worked up over it. which, while i wouldn't phrase it in the way the parent did, i agree with the direction of their point.

Re:Incident with Github.com by sarjann on Monday August 17, 2026 @10:35AM (#49331731)
Except he tries justifying it.

"Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code"

"Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside." with assumptions like above.

If it was, "hey they f*ed up but there's no point having an overly emotional reaction" that would be fine. But he seems to be justifying this. Especially with Github's record up to now of unreliability I think it's completely fair to be annoyed.

Re:Incident with Github.com by zahlman on Monday August 17, 2026 @08:18PM (#49339523)
I don't even see a whole lot of emotional worked-up-ness, though. Just continuing disappointment.
Re:Incident with Github.com by madeforhnyo on Monday August 17, 2026 @10:10AM (#49331264)
> due to LLM sloppers pushing large amounts of trash code

Which they encouraged by pushing Copilot down everyone's throat

Re:Incident with Github.com by constant_flux on Monday August 17, 2026 @10:15AM (#49331366)
GH shouldn't have promoted LLM usage the way they have if they didn't have the infrastructure to support it. Regardless of how you feel about the code quality (which you have zero evidence of), if GH has made commitments to supporting broad LLM usage, they need to back that up with the proper hardware and without whatever fragile SDLC processes they have.
Re:Incident with Github.com by pamcake on Monday August 17, 2026 @10:27AM (#49331585)
> GH shouldn't have promoted LLM usage the way they have

No qualifiers necessary. But try arguing with a lawnmower...

Re:Incident with Github.com by bhouston on Monday August 17, 2026 @10:06AM (#49331198)
> To everyone who is angry: calm down. Github’s servers are constantly on fire

Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?

Re:Incident with Github.com by s_dev on Monday August 17, 2026 @10:03AM (#49331125)
What if we're not angry and just perplexed? Do we calm down further or can we still ask questions?
Re:Incident with Github.com by ethagnawl on Monday August 17, 2026 @10:37AM (#49331753)
No questions. Only bench press.
Re:Incident with Github.com by armSixtyFour on Monday August 17, 2026 @11:50AM (#49332983)
Hard disagree, people are paying GitHub a lot of money.

If they are facing an influx of commits from bots maybe they need to start metering commits from those accounts and charge them money.

This is a self imposed problem.

Re:Incident with Github.com by malfist on Monday August 17, 2026 @10:23AM (#49331504)
Why do you get to tell me how I should feel?
Re:Incident with Github.com by _joel on Monday August 17, 2026 @10:31AM (#49331641)
It's every other day now. We pay for a service, we expect service.
Re:Incident with Github.com by AgentK20 on Monday August 17, 2026 @10:01AM (#49331080)
what if I do have an emergency hotfix for a production outage though ;-;
Re:Incident with Github.com by DharmaPolice on Monday August 17, 2026 @10:15AM (#49331374)
This might be naïve but wouldn't the appropriate response be to reduce access/rate limit free/new accounts in order for service to be maintained for everyone else?
Re:Incident with Github.com by stetrain on Monday August 17, 2026 @10:15AM (#49331373)
> as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

Github and their parent company are active participants in pushing LLM-driven coding.

Re:Incident with Github.com by phtrivier on Monday August 17, 2026 @10:05AM (#49331180)
I understand the need to level-head the discussion and calming everyone.

However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow.

The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available.

(I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)

Re:Incident with Github.com by fn-mote on Monday August 17, 2026 @11:58AM (#49333122)
It's past time to move your critical infrastructure off Github then, no?
Re:Incident with Github.com by phtrivier on Monday August 17, 2026 @12:56PM (#49334105)
It's not exactly done in a pinch, but yeah, I guess someone must be working on disrupting CIs by the audacious value proposition of "we'll work when you need to".
Re:Incident with Github.com by maccard on Tuesday August 18, 2026 @01:45PM (#49349527)
That effectively means “stop using GitHub” though. Which is definitely worth considering but also kind of insane to think as the best option.
Re:Incident with Github.com by anonymousab on Monday August 17, 2026 @10:12AM (#49331317)
Allowing and accepting the LLM load is a willful choice they made and are making at the expense of their users, including their paying enterprise users.

They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.

Re:Incident with Github.com by pamcake on Monday August 17, 2026 @10:19AM (#49331440)
Allowing and accepting? More like encouraging and instigating.
Re:Incident with Github.com by robofanatic on Monday August 17, 2026 @10:41AM (#49331853)
> constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

I find it hard to believe that github APIs don't have rate limits to handle traffic spikes.

Re:Incident with Github.com by kilroy123 on Monday August 17, 2026 @10:24AM (#49331523)
As a _paying_ customer, I disagree.
Re:Incident with Github.com by AustinDev on Monday August 17, 2026 @10:11AM (#49331282)
Alternatively, don't use GH for critical infrastructure they are no longer reliable, simple as.
Re:Incident with Github.com by slowin on Monday August 17, 2026 @10:10AM (#49331275)
Well, they’re a critical piece of infrastructure with terrible stability. I’m in the process of migrating us off GitHub now.

I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.

Re:Incident with Github.com by raincole on Monday August 17, 2026 @10:02AM (#49331094)
Or, you know, it's perfectly reasonable and natural to feel angry when a service you paid for gets worse over time.

> Unless you have a emergency hotfix (you don’t)

Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.

Re:Incident with Github.com by javier2 on Monday August 17, 2026 @10:05AM (#49331165)
While I agree in principle, Github with their CI is very critical to many parties.
Re:Incident with Github.com by semiquaver on Monday August 17, 2026 @10:26AM (#49331566)
Why would you take what they say at face value? I believe that this is basically a lie covering up deeper rot.
Re:Incident with Github.com by Topfi on Monday August 17, 2026 @10:03AM (#49331132)
No one ever has to commit anything time sensitive.
Re:Incident with Github.com by delduca on Monday August 17, 2026 @10:00AM (#49331068)
> increased something like 50x due to LLM

This should not be an excuse since they probably can use AI to fix it /partially sarcasm

Re:Incident with Github.com by bradhe on Monday August 17, 2026 @10:39AM (#49331791)
> due to LLM sloppers pushing large amounts of trash code

hey bud your bias is showing

Re:Incident with Github.com by thejosh on Monday August 17, 2026 @10:44AM (#49331898)
it's been trash for years, their status page just doesn't reflect that.
Re:Incident with Github.com by OtomotO on Monday August 17, 2026 @10:21AM (#49331481)
You know, I am not employed.

I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket.

So now I am not angry because some critical piece of US-american infrastructure is down all the time.

I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.

Re:Incident with Github.com by nerdix on Monday August 17, 2026 @10:17AM (#49331397)
Wouldn't be surprised if we lost free private repos because of the slopocalypse. Though most don't seem to be shy about sharing their slop with the world so I'm not sure if that will actually increase stability.
Re:Incident with Github.com by c2403 on Monday August 17, 2026 @10:08AM (#49331224)
In this case, perhaps they shouldn't allow AI slop or promote usage of AI agents...
Re:Incident with Github.com by cesarb on Monday August 17, 2026 @10:04AM (#49331158)
> go hit the gym or walk outside.

My mind immediately went to this classic xkcd: https://xkcd.com/303/

Re:Incident with Github.com by catigula on Monday August 17, 2026 @10:27AM (#49331586)
FYI this is an argument to not use Github, not an argument to use it.
Re:Incident with Github.com by sharts on Monday August 17, 2026 @10:19AM (#49331443)
It has nothing to do with load.
Re:Incident with Github.com by otterley on Monday August 17, 2026 @10:24AM (#49331519)
Re:Incident with Github.com by karissa on Monday August 17, 2026 @10:56AM (#49332103)
Re:Incident with Github.com by otterley on Monday August 17, 2026 @11:54AM (#49333053)
First, that source is unreliable, particularly for before the acquisition. (We don’t know whether the data is accurate, it’s based on GitHub’s own self reporting.)

Secondly, the scale is WAY off and makes it look far worse than it is. It makes it look like 99.5% availability is practically zero availability.

Finally, GitHub’s availability is not an all-or-nothing proposition. They report incidents on a granular level, for example, webhook firing can be impaired while Git hosting may be working fine.

Re:Incident with Github.com by maccard on Tuesday August 18, 2026 @01:50PM (#49349602)
0.5% downtime is an hour a week. If that happens for 10 minutes a day at 11pm PT it doesn’t matter, if it happens for half a working day once a month, it’s a disaster.
Re:Incident with Github.com by dimgl on Monday August 17, 2026 @10:05AM (#49331174)
What if you, you know, have a job?
Re:Incident with Github.com by TZubiri on Monday August 17, 2026 @10:11AM (#49331292)
is the infrastructure of the free tier shared with that of paid users? That might be the issue. Otherwise if usage of paid users scales, then it would be fine.
Re:Incident with Github.com by john_strinlai on Monday August 17, 2026 @10:16AM (#49331378)
>is the infrastructure of the free tier shared with that of paid users?

github enterprise is fully operational at the moment

Re:Incident with Github.com by nope1000 on Monday August 17, 2026 @10:24AM (#49331525)
At least enterprise EU is partially affected

https://eu.githubstatus.com/posts/dashboard

We cannot pull any actions images (or how it is called) for example.

Re:Incident with Github.com by neo_doom on Monday August 17, 2026 @10:12AM (#49331300)
Agreed. This impatience culture is so toxic. Been down less than 30 mins.
Re:Incident with Github.com by anonymars on Monday August 17, 2026 @10:43AM (#49331870)
That's cool, I mean if you'd gotten here earlier you could have said "been down less than 5 minutes"

So with the "elapsed time" out of the way (now over an hour, incidentally), how about "how long will it be down?"

Re:Incident with Github.com by neo_doom on Monday August 17, 2026 @10:24PM (#49340409)
Ended up being down for about 4 hours. Hope you were able to get out and touch grass.
Re:Incident with Github.com by anonymars on Monday August 17, 2026 @11:06PM (#49340773)
For some people losing half a day of work is kind of important. Hope you were able to get out and find some empathy

https://news.ycombinator.com/item?id=49331481

Re:Incident with Github.com by cb33 on Monday August 17, 2026 @12:18PM (#49333486)
2 hours later - still down. Is this "impatience culture" or incompetence culture at this point?
Re:Incident with Github.com by bmitc on Monday August 17, 2026 @01:19PM (#49334458)
If we all go to the gym every time GitHub is down, we're all gonna get ripped. GitHub was down all day just last week.

They get paid millions of dollars by my company. They need to get it together.

Re:Incident with Github.com by bluerooibos on Monday August 17, 2026 @03:15PM (#49336177)
> calm down

If people and companies are paying for critical infra they expect it to be available - it's not MY problem that their servers are on fire - last time I checked, having too much demand was a good problem to have and they've had over a year to fix it and come up with a plan.

I'm so glad I migrated to GitLab months ago.