But there are so many hooks into the Git workflow today, and I’m struggling to find a “user story” that would not be met by custom Git tooling. UX could be added to Git objects à la Gerrit with custom CLIs, custom Gitoxide ODB and (eventually) ref storage backends can help the server scale, and things like promisor remotes and partial clones can keep things smaller on the client side.
I’m curious what ERSC is able to share publicly about where the scaling fails with the Git specification? Is it the wire format, or the relative speed of upstreaming new things to Git? Or is the side-channel approach for storing rich data in Git objects insufficient in some way?
P.S. I wrote a blog post[1] about this a bit ago. I’m sure I’m missing something. Since I wrote it, I’ve wondered if the lack of deferred conflicts in-tree would be a scaling limiter.
Yes, it's important to realize that git is multiple things: the on-disk repo format, the CLI, and the wire protocol. Each has their own challenges, just like any project, but they can be tackled independently. But at the end of the day, there just are fundamental limits to the way that git does things that make total sense in the context Linus developed git for, but just do not in other ones. Some of those are performance related, but some of those are things like how the kernel is an open source project, and so per-directory ACLs are something that git wouldn't ever bother supporting, but are valuable for companies. We'll be talking about these things in the future, so I'm just going to be hand-wavy about it right now. But we do think that git is a really great system that has served us all well for 20 years, but just like how git displaced systems before it due to significantly re-thinking the details, for some users, making meaningful improvements over git will also require a similar significant re-think. That's just the way of technological development.
I’m not saying not to try or that there’s no value in turning the wheel once more. But I used to be excited about these kinds of things and now I’m not because I’ve seen the cycle a few times now.
You mention performance issues with Git, but there are performance issues with this blog post.
No one has agreed to this premise. You have invented it out of nowhere.
Normal Git/JJ is fine for most people, and most open source projects. It's just when you get to the scale of commercial software companies (100+ employees) you run into the classic monorepo/submodule problem.
Monorepos are currently the best solution - they make testing tractable, you can do cross-project changes, Git's support for them isn't buggy as hell. But they have obvious downsides - performance is the big one, but also integration with open source projects, permission control, etc.
So there's this big problem space that really only matters to companies where as far as I know there are currently no good open source or even commercial solutions. The only people that have (maybe) solved it are the mega-corps like Facebook and Google who have made their own private solutions. I don't know exactly what ERSC are going to offer but it's surely going to be in that space and also probably not going to be super relevant to private individuals.
Our moat is that we are not trying to take git and scale it up, we are building a more scalable system from the beginning. And we will have features that nobody else has because of it.
It's always the case that businesses have competitors, that's just capitalism. We're not worried about that.
But that's good to hear.
Initially we won't be offering our own CI directly, you'll bring your own. We use Buildkite internally, so we'll probably share that integration first, but fundamentally you can do whatever you'd like. We won't require you to use whatever first-party offering we end up with.
While jj works with git, it doesn't have to. jj has the ability to work with many backends. Git is the popular one in the open source world, but that doesn't mean that other ones are not possible. And that's the core tech we have built.
With the rise of agents, more and more companies are running into the limits of what Git can do. It was never designed for large monorepos, and so there are performance issues, but also features that it's missing, that are important once you start trying to scale things up to an entire organization. Companies like Meta and Google ran into these things well before the present moment, but you don't have to be a FAANG these days to be running into this sort of stuff.
We also understand that incremental adoption is really important, especially for something as fundamental as source control. So our tech also has a git ingress/egress that will work with the core functionality, but more advanced things that git simply can't do will end up requiring using a non-git protocol. jj is the bridge between those two worlds.
I hope that makes sense! We have been in psudeo-stealth for a while now, this is the start of us coming out of our shell a bit and actually talking about what we're doing.