Re:Pijul 1.0 Beta
by WorldMaker on Wednesday January 19, 2022 @03:07PM
(#29999266)
Git has hit a strong "local maximum" of "good enough". If you like git, there's probably no reason to move to something better, because it is a comfortable "local maximum" (especially with the network effects of everyone now learning/using git and git just being the comfortable de facto winner).
Everyone here points out that the big advantage to something like pijul is in how it does merges. Git has a set of extremely complicated merge algorithms (the most common up to today has been "recursive merge" which you'll see mentioned a lot in git console output, it's on the way to being replaced with ORT [Ostensibly Recursive's Twin] which is a from-scratch rewrite of the "recursive merge" algorithm with a better understanding of the problem space; there are a couple other algorithms [strategies in git parlance] that have more niche uses). When they work well, they do a brilliant job, but they are extremely complex algorithms and have to do a lot of work to figure out things like "what things got renamed/moved and where between these two branches?". Git mostly doesn't store any of that information at all and generally computes it all from scratch every time it is needed. (You'll notice, again, if you watch console output a lot how often "rename detection" alone often runs, sometimes on the same commits over and over.) But a patch language like pijul encodes things like renames/moves much more directly and doesn't need complex algorithms to guess when they occurred because it wants the user to encode that more directly (as possibly a better "higher level" representation of the user's intent: the user wanted a file renamed/moved and that was an action they took).
In theory then, the merge algorithm of a patch language like pijul (or darcs, its older relative) is simpler because it has a lot more higher level recorded constructs and a lot less "guesswork heuristics" to try to oracle user intent (sometimes long after the fact). (In practice we find such merge algorithms have nearly as much complexity in their own way. The patch algebras of pijul and darcs are both academically related to work done on OTs and CRDTs, all three approaches informing each other, which is in part why darcs was written in Haskell. [pijul is not, just as we have CRDT libraries in most languages today, we've come a long way in our understanding of the complexities of these tools since the early days of darcs.])
Though, I don't think merges on their own are the killer thing that makes an approach like pijul's truly better than git's. Again, git has achieved an incredible "local maximum" of "good enough". As complex as git's recursive and ORT merge strategies are, they are by far "good enough" (and hands down better than many predecessor VCSes) with a lot of smart engineering work behind them at this point. In many "day to day" uses cases you aren't going to notice a huge difference from git's well optimized "dumb" "guess what the user was thinking at the time" merge algorithms and pijul's smart "capture what the user was thinking at the time" approach. Most people don't see the horrors of things like git's rerere cache in their day-to-day git lives (and that's a great thing; I would not recommend learning git rerere if you can avoid it).
The killer feature is cherry picking. Git has git cherry-pick and it "exists" technically, but anyone who is smart has warned you away from ever using it, and especially not in day-to-day workflows. Cherry Picking is the concept of taking just one or two commits from the middle of a branch and applying them to a different branch without taking the other changes. In git this creates entirely new commits with their own very different identities that are entirely unrelated on the DAG (git object graph) in any way to the original commits. It's like a rebase, but much much worse because typically you rebase an entire branch and throwaway the originals, but if you are desperate enough to cherry pick you likely need to keep the originals as well safe in that other branch. In my experience in git this is one of the worst sources of bad merges when those two branches eventually (and often inevitably) merge. A lot of the "dumb" heuristics in git's merge algorithms get really dumb when the same changes were made in both branches.
On the flipside, cherry-picking is a "native" behavior of a patch algebra and how its merges work. Just like in CRDTs that are eventually consistent, a patch is often the "same" no matter what branch it is in (context it has of other patches/time it merges in on other devices in a CRDT perspective). You can generally pull an individual patch between branches with ease whether "beginning, middle, or end" of that branch, and it will just work. (And when you eventually merge branches back, though it is far less "inevitable" than in the git case, it knows the same patch was already applied once and doesn't have additional work to do.)
When I was using darcs heavily, I used a lot fewer branches than in my workflows in git. I knew I could pretty easily cherry pick changes between branches without worrying about where they were in "commit order", so often individual patches felt like entire branches (or tags) and you could often pick and choose specifically what you need at any time. Cherry picking was so common it was just assumed you could do it. (There are cases where you couldn't quite cherry pick what you want, where patch has a more direct dependency on an earlier patch that you didn't expect, but the darcs UI was really good about making it clear that pulling one patch would pull some others it relied on.)
Pijul should be the same, "natively and easily cherry picking", and that can be a killer feature that git doesn't have a lot of ways (today) to do better.
In many ways that "native cherry-picking" felt like a much better way to flow changes between branches. Cherry picking always seems like a key feature that developers need (often because management needs it: "can you get just Feature X into Production without Feature Y because Feature Y isn't ready yet?" but they were developed/integrated in the same branch), if not how they naturally think of changes flowing between branches, and I've seen too many teams already fall to "well git cherry-pick exists so it might help us here" and the ugly merge hell that approach leads to (including horrors like hand-holding the git rerere cache).
Everyone here points out that the big advantage to something like pijul is in how it does merges. Git has a set of extremely complicated merge algorithms (the most common up to today has been "recursive merge" which you'll see mentioned a lot in git console output, it's on the way to being replaced with ORT [Ostensibly Recursive's Twin] which is a from-scratch rewrite of the "recursive merge" algorithm with a better understanding of the problem space; there are a couple other algorithms [strategies in git parlance] that have more niche uses). When they work well, they do a brilliant job, but they are extremely complex algorithms and have to do a lot of work to figure out things like "what things got renamed/moved and where between these two branches?". Git mostly doesn't store any of that information at all and generally computes it all from scratch every time it is needed. (You'll notice, again, if you watch console output a lot how often "rename detection" alone often runs, sometimes on the same commits over and over.) But a patch language like pijul encodes things like renames/moves much more directly and doesn't need complex algorithms to guess when they occurred because it wants the user to encode that more directly (as possibly a better "higher level" representation of the user's intent: the user wanted a file renamed/moved and that was an action they took).
In theory then, the merge algorithm of a patch language like pijul (or darcs, its older relative) is simpler because it has a lot more higher level recorded constructs and a lot less "guesswork heuristics" to try to oracle user intent (sometimes long after the fact). (In practice we find such merge algorithms have nearly as much complexity in their own way. The patch algebras of pijul and darcs are both academically related to work done on OTs and CRDTs, all three approaches informing each other, which is in part why darcs was written in Haskell. [pijul is not, just as we have CRDT libraries in most languages today, we've come a long way in our understanding of the complexities of these tools since the early days of darcs.])
Though, I don't think merges on their own are the killer thing that makes an approach like pijul's truly better than git's. Again, git has achieved an incredible "local maximum" of "good enough". As complex as git's recursive and ORT merge strategies are, they are by far "good enough" (and hands down better than many predecessor VCSes) with a lot of smart engineering work behind them at this point. In many "day to day" uses cases you aren't going to notice a huge difference from git's well optimized "dumb" "guess what the user was thinking at the time" merge algorithms and pijul's smart "capture what the user was thinking at the time" approach. Most people don't see the horrors of things like git's rerere cache in their day-to-day git lives (and that's a great thing; I would not recommend learning git rerere if you can avoid it).
The killer feature is cherry picking. Git has git cherry-pick and it "exists" technically, but anyone who is smart has warned you away from ever using it, and especially not in day-to-day workflows. Cherry Picking is the concept of taking just one or two commits from the middle of a branch and applying them to a different branch without taking the other changes. In git this creates entirely new commits with their own very different identities that are entirely unrelated on the DAG (git object graph) in any way to the original commits. It's like a rebase, but much much worse because typically you rebase an entire branch and throwaway the originals, but if you are desperate enough to cherry pick you likely need to keep the originals as well safe in that other branch. In my experience in git this is one of the worst sources of bad merges when those two branches eventually (and often inevitably) merge. A lot of the "dumb" heuristics in git's merge algorithms get really dumb when the same changes were made in both branches.
On the flipside, cherry-picking is a "native" behavior of a patch algebra and how its merges work. Just like in CRDTs that are eventually consistent, a patch is often the "same" no matter what branch it is in (context it has of other patches/time it merges in on other devices in a CRDT perspective). You can generally pull an individual patch between branches with ease whether "beginning, middle, or end" of that branch, and it will just work. (And when you eventually merge branches back, though it is far less "inevitable" than in the git case, it knows the same patch was already applied once and doesn't have additional work to do.)
When I was using darcs heavily, I used a lot fewer branches than in my workflows in git. I knew I could pretty easily cherry pick changes between branches without worrying about where they were in "commit order", so often individual patches felt like entire branches (or tags) and you could often pick and choose specifically what you need at any time. Cherry picking was so common it was just assumed you could do it. (There are cases where you couldn't quite cherry pick what you want, where patch has a more direct dependency on an earlier patch that you didn't expect, but the darcs UI was really good about making it clear that pulling one patch would pull some others it relied on.)
Pijul should be the same, "natively and easily cherry picking", and that can be a killer feature that git doesn't have a lot of ways (today) to do better.
In many ways that "native cherry-picking" felt like a much better way to flow changes between branches. Cherry picking always seems like a key feature that developers need (often because management needs it: "can you get just Feature X into Production without Feature Y because Feature Y isn't ready yet?" but they were developed/integrated in the same branch), if not how they naturally think of changes flowing between branches, and I've seen too many teams already fall to "well git cherry-pick exists so it might help us here" and the ugly merge hell that approach leads to (including horrors like hand-holding the git rerere cache).