As if more developers means better code. Wasn't it Bonhoeffer who said stupidity is when smart people don't question authority?

So that is why I got a bunch of 500's during push this week :)

Hope it helps improve reliability, it was getting hard to defend GitHub with all the recent outages.

If I'm reading this right, this is (really solid) future planning but won't help with the current outages?

I'd have thought so from the headline + initial content but then they say that the horizontal scale techniques are basically carrying over from the existing architecture (where it seems the problem is?) and this is mostly to help with what they call "vertical scale" (intra-repo scalability). They also admit right after that the most active repos today will receive short bursts of at most dozens of writes.

So this seems like really useful planning for a near future that likely helps prevent more issues but isn't really meant to help current issues. Am I reading this right?

Of course improvements like this are always useful either way. Not dunking here at all.

Since they wouldn't need to actually have the inactive repositories on disk anymore, I reckon it should solve the horizontal scale issue too by reducing the compute/disk capacity that needs to be maintained.

Disclaimer: I am building a code forge myself at https://juju.bi, so I am definitely not biased.

personally I see more issues with actions over pushing

What? 7.2k CI runs per second is a piece of cake! ;-)

the ci problem I think is fundamentally easier, since each is its own instance, you just need an orcestrator to coordinate, that can also be one per a group of repositories (there are no cross-repo triggers anyway). The actions issues we keep facing seem to be at the agregation/collector side of things, similar to general api issues, not a fundamental kubernetes scaling issue.

So we can have one more 9?

It will be nice having maybe a few at the beginning of the number!

Maybe every other month.

They already give us one, you want a _second_? Entitled behaviour

Can't they implement some kind of read-only mode?

93.999

Most of this is similar to what https://enroute.sh already did scaling Git. https://enroute.sh/blog/everyone-is-doing-git-on-s3-wrong

At this point GitHub should likely just try snatching the Enroute guys :)

Looks cool. Anybody here with experience to share?

N.b the slow sun rise with the parallaxing hills on scroll is a great bit of 2000's-style CSS.

It might be more of a send up to Firewatch these days: https://www.firewatchgame.com/

But I'm still waiting on someone to revive 2008's darling, resizing parallaxing: https://web.archive.org/web/20080302013806/https://www.think...

If you like a CSS sunrise over some hills, you'll probably like my friend's website [1] and write-up [2].

[1] https://grady.dev/

[2] https://grady.dev/projects/sunrise/

The end result has hardcore color banding but I love how much thought went into this.

The old-school way to deal with banding is with noise, I wonder if a semitransparent overlay of white noise would help here. Thanks for the link!

Sorry, but thousands of pushes per second, per repository?! In what world could that possibly seem like a sound idea?

Sure, it's nice that you can technically handle it, but to me it sounds like at that point something has gone absolutely horribly wrong in the way you set up your agents. Last time I checked, GitHub was intended as a platform to upstream finished work items, not a realtime cross-agent chat channel.

Handling thousands of git pushes per second implies thousands of issues, thousands of PRs, tens of thousands of comments, thousands of Actions runs, and dozens of releases. How's that part of their infra going to scale?

How about introducing rate limits instead, nudging people to not have agents push every individual single-line changes to GH, but instead have them collaborate as a local cluster and only upstreaming PR-sized chunks of finished work?

If you still need thousands of pushes per second you'd be looking at millions of lines of code, and at that point you're looking at an insane monorepo which is growing at a rate of terabytes per day, and which basically instantly becomes unusable due to being impossible to clone or check out.

My understanding is that forks on GitHub share the same storage in the GH backend.

It's not unreasonable to imagine thousands of commits per second to the kernel, especially if you consider that bots might be watching LKML and applying patches to various forks.

Welcome to the new AI future, where everyone is insane and reality doesn't matter.

I'm starting to think AI isn't a productivity machine, but rather a machine for hypnotising people into the false belief of productivity.

Yesterday's "web scale" is today's "agentic workload"

You weren't supposed to notice

That's exactly my thoughts! AI generated code is not good code, it's just engineered to look good

How do people afford such costs though? At that rate, you're not running flagship models due to cost and limits.. right..?

Yeah it’s just total slop. If your code is slop anyway, why commit it? Just push it at major milestones. It’s not like it matters if you lose it.

> Sorry, but thousands of pushes per second, per repository?! In what world could that possibly seem like a sound idea?

The only conceivable scenario that I can imagine leading to this volume of writes is a push-based GitOps repository for a huge cluster holding a large volume of apps, each one with many services. Each commit in one if these services triggers a commit to the gitops repo to update the reference to the new container image, which in turns triggers a deployment update.

There is no software project on earth that justifies a rate of changes in magnitudes of dozen of commits per second. This is a symptom of a huge problem, not a sign of times. Even in monorepo mania this sort of thing is inconceivable.

But that's also what you get with vibecoding: lots of output from people who do not have a clue about what they are doing.

You basically described Google’s monorepo… about 10 years ago.

Companies pay GitHub for enterprise repositories. It’s not completely wild that a 2026 large scale company would have the same scale as 2016 Google…

A different way of looking at this is that it gives GitHub a huge amount of headroom. The architecture they're describing sounds more efficient at lower rates too, it's not like better throughput and efficiency only benefit repos that are reaching the maximum limits.

Plenty of people (and agents), me included, use commit-and-push as an equivalent to "Save". You spill coffee on a day of unpushed work once, you stop caring about dogma like "commits should be finished, testable, reviewable units of work with discrete, human-readable descriptions" and start treating your Git remote as your backup system.

Besides, you can push amends to your branch, or describe/review/squash your changes at PR merge time anyway, so what's the harm of a zillion pushes? Other than "GitHub engineers have to invent a novel database", that is. But that's probably enrichment for them.

You could also just ban people who spam you and consume immense amounts of server time. Or make them pay for the server time.

it’s pretty ironic we got all the complexity of distributed source control with all of the drawbacks of centralisation

I don't think this is ironic at all. The source control system is decentralized, in the sense that you can checkout a repo and use it as the central source or truth for everything.

For example, you can clone a repo from GitHub into your workstation,and then clone that repo into multiple folders in your workstation where you unleash agents to implement changes. The you pull those changes to your local repo or even directly to GitHub or GitLab or whatever. That's pretty decentralized.

Now, compare this to the god awful mess that was subversion.

For god's sake, the git workflow you've just described is centralized. It has nothing to do with decentralization in practice.

GitHub is still a single source of truth in that scenario.

BTW, it's unclear what exactly you think is wrong with SVN. Works great when used properly (like any tool, you know).

Agent scale is definitely on the mind around industry, I wonder what github folks would say in response to https://www.anthropic.com/research/redesigning-source-contro.... It's an interesting read to step back and consider source control from first principles and rebalance the tradeoffs for what qualities matter in this new regime.

Thousands of changes per day means no human reads all these changes.

There're really security issues, and a black box creates black boxes.

Humans are also black boxes

>> Humans are also black boxes

I have no idea why I like this comment, but I do.

There are not, because humans are where the light shines out of in this analogy.

Have you met a human before? Are they totally arbitrary? If not they are not black boxes. I’m getting a bit tired of this equivocation. We are all 99.999% the same.

There is no "industry" anymore, and "Claude prompter" is not a job description.

[delayed]

Possibly Shopify's monolith.

Or alternatively someone is doing something very silly.

Nah, they replaced GitHub with some in-house videcoded stuff a few months ago, and from what I hear from the grapevine it's not going well.

But there's a number of other big private repos with a high rate of merge on GitHub, and indeed it's currently a bit painful, e.g. the subroutine that check whether a PR can be merged tend to take ages. So my read is that they want to better support these big customers out of concern that they might leave too.

This is just cringeworthy, even for Tobi.

The biggest very active repo I know of is nixpkgs, and I heard it was a stress test for GitHub already.

Possibly this dude's repos? https://github.com/xesf

Every day, he sends hundreds of commits to his (most likely vibecoded) repositories of retro games recompilation.

[delayed]

It's probably a horror story if we ever find out.

Given its prominence in some of these other discussions from GitHub of a repo always on their performance radar now my gut instinct is that it could be the Windows monorepo.

Windows? We know it's on Git, and Microsoft owning GitHub means it would make sense for it to be on GitHub.

Alternatively, something from IBM with 60 years of commit history.

Based on Azure... Isn't that one of the more unreliable clous in the market?

Define unreliable? Azure has a 99.97% historical uptime, AWS at 99.95%. Azure is #2 in market share, right beyond AWS. Engineer anecdotes about internal strife doesn’t equate to external reliability. I’m sure you can find AWS engineers complaining about AWS, or GCP, or insert company. (Also, Microsoft owns Azure and GitHub, so what did you expect?)

It was a question, not a statement. Yes, you can find complaints about every company. I have never used Azure, but remember people complaining like in a sibling comment. I have used AWS for 15 years, some years even a lot. It's mostly working, although I also have my complaints. They have billed me for a VM after sending an email that it has probably failed. And yes, it was not working correctly. As we speak they have locked me out of my private account, not particularly important, and support can't tell me why. They are still billing. Nothing really surprising, the cloud is not your computer. And hyperscalers have no customer support worth the name unless you pay $$$.

What do those uptime numbers tell? For 95% of the problems I had the status page has no information.

Where and how are you getting that reliability data? And what is it for? All services? A region? Everything averaged out? I know companies embarking on multi million dollar projects to add a backup identity provider to EntraID because it's that unreliable and it costs them money every time it's not working.

Azure was the cloud that had all global auth rely on a single DC in Texas (we learned when cooling failed there), and didn't have availability zones in most regions until not that long ago. They've also had a number of trivial cross tenant exploits. No, two us-east-1 outages that had minal global impact isn't comparable, we're talking about all auth for all of Azure using a single DC, not a region of 5 DCs having the control plane for global AWS services.

Neither security nor reliability are a priority. It's by far the worst big cloud provider, and it's not even remotely close.

Azure is most unreliable for me because of how terrible the UI is, plus ipv6[0]. Agents translating natural language queries/actions to Azure api calls is a lifesaver.

0: https://news.ycombinator.com/item?id=29327773

I wonder if they’re working on a non git version forward looking for agents as well

This new brilliant idea that agents need something different/better than git is simply ridiculous.

The problems of scale we're encountering are because bad developers are deploying bad quantities of bad agents against bad repos.

Vibe-coded repos occupy GB of space, utterly useless PRs are opened and closed due to conflicts with other useless PRs, and it turns out git is the problem!

I thought Actions (for free repos) was the main culprit, no?

You're not wrong, but GitHub can only control what they can control. They cannot hold everyone at gun point and make them stop vibe coding. They also cannot shutdown Anthropic/OpenAI - you'll have to wait for their A/OAI's debts to come due for that one.

We might see real usage limits come to GH's free plan. TBH, I bet they're regretting their decision to allow unlimited private repos on the free plan, at least a bit.

Afaik all their problems are caused by public repos, private are not that expensive.

I think humans need something better than git :)

Once they get it, agents will get it too! Eg. I'd switch over to bazaar/breezy with comparable performance to git — but mostly comparable adoption — right away.

I think perforce might be the better VCS if you intend to have a bunch of autonomous agents attempt to collaborate over one big pile.

Multiple terabytes is no problem out of the box. You get exclusive locking of files. You can easily take partial checkouts of the monolith. Centralization is a feature in this particular context. Git works great if the need for coordination is moderate. If you've got thousands of automated developers with their hands in the cookie jar, exclusive locks start to look really compelling. Perforce sucks with a highly distributed team, but in the case of an AI agent swarm we presumably have the ability to instantly nuke an agent's checkout if needed. I don't have to contact the agent and wait for them to get out of bed.

Perforce is not bad, but using exclusive locking is just bad practice, for both humans and agents. Especially for agents. Agents are good at resolving merge conflicts.

This looks a lot like Cursor's Continuity, though Github are reporting an order of magnitude greater write throughput.

https://cursor.com/blog/git-at-any-scale

"we’ve been building it underneath GitHub while it keeps running."

Holding on for dear life and failing is more like it.

Thanks for the downvotes. Github is down today by the way.

> sending 7,200 pushes per second to a repo

Imagine the token costs!

My company with about 60 devs in the backend dev slack channel probably produces less commits per year together.

Does my company already belong to the permanent AI underclass?

A manager somewhere will read this and require similar output from their team.

Huh ?

I commit that amount per year (alone, no AI, dev is not my main work)

Am I the only one reading this with one raised eyebrow, wondering if a team of architecture astronauts have over-engineered this just a bit?

Pretty much everything in Git can be abstracted away as “immutable blob that can be conveniently content-indexed” and an “event log”. Both are embarrassingly parallelisable since repos are independent, and every blob is independent!

Caching is trivial over immutable blobs, and they would just need to partition the event log store (Kafka or whatever) so that each repo is maps to one partition.

Kafka can sink, what, a million events per partition per second?

Similarly, LSM log and KV storage like Microsoft FASTER would let them scale even that to about a hundred million events per sec, far beyond their target!

Am I missing something?

i wonder how much traffic can be avoided by pushing for bandwidth-considering updates to the git client itself.

things such as shallow clone by default, applying lazy evaluations on repo with large commit volume (e.g. partial remote-related operations at branch level until it is switched to), etc.

if reading these overloaded graphs is the problem, most readers (nowadays through automated consumption) only care about the current snapshot of a project's code. that too in probably the main branch.

github can probably make the git maintainers implement this. feels like it is worth trying at least.