If I were to pie in the sky dream up a replacement for git, I'd have it store the AST of the parsed code instead of a text file. It would solve a lot of problems with refactoring crapping all over the history. Like I said, pie in the sky. Probably never gonna happen.

Personally I don't see git's problems with large binary files and tens of millions of commits as being major issues. Those two alone are way less valuable than git's ecosystem and mindshare.

What you want has nothing to do with git (which is a storage model). You can use arbitrary diff and merge resolution algorithms with git's plumbing, which would give you the AST-aware functionality that you want.

I used VisualAge's Envy for 3 years while working in a Smalltalk project.

Envy does versioning of classes and methods, and you can programmatically access to the model.

It allowed us to build tools around the VCS. For example, we had tools to merge multiple feature branches and resolve conflicts automatically. We also used the same tools to produce migration scripts for our database (GemStone). That was 18yrs ago! and today sounds irreal.

You can build tools on top of git, but the versioning “unit” gets in the way. (e.g imagine the possibility to encode refactorings in your change history and reapply or rollback them).

I’m not trying to criticize git. I think it is the best file based DVCS. My point is that many dev tools that we use today are extremely rudimentary, because the lack of good abstractions. And I don’t think that git provides a good model to build those abstractions on top of it.

That's like saying DVCS has nothing to do with VCS -- it's just the server model. Every major advance has been accomplished by increasing the scope of version control.

Arbitrary diff/merge in Git is a great example of the Turing Tar-Pit. It's possible, but prohibitively inefficient for many things I want to do. You can't add your own types, index, or query optimizations.

Today, if I want to store data for my application, I have a choice between good support for rich object types and connections (e.g., Postgres), or good support for history and merging (e.g., Git). There's no one system that provides both.

> Today, if I want to store data for my application, I have a choice between good support for rich object types and connections (e.g., Postgres), or good support for history and merging (e.g., Git). There's no one system that provides both.

I like the way you put this. In case anyone's interested in brainstorming I'm dabbling in this problem with a thing called TreeBase (https://jtree.treenotation.org/treeBase/). It's still a toy at this point, but it stores richly typed data as plain text files to leverage git for history and merging and then can use SQLite (or others) for querying and analysis. A very simple database in the wild looks like this: https://github.com/treenotation/jtree/tree/master/treeBase/p...

Have you looked at Qri? (https://github.com/qri-io) - free & open source dataset versioning. Also: https://qri.io

AFAIK I've never seen that one. Thank you very much for the link. Looks very interesting and related to the stuff I'm working on. Thanks!

Couldn't you store the exported database as sql commands? I'm not familiar with every git hook, but if there aren't enough to automated that I guess you could wrap it.

The slowness of destroying a whole database and then recreate it when checking out should be something you can handle by relying on the diff to generate a series of delete commands and a series of insert commands.

But yeah, I guess committing will be slow if you have a lot of data to export. For the time being, it's a trade off to be made.

[I might consider testing this with my current database project. But I'm using SQLite so I guess that implies a lot less data than Postgres.]

> That's like saying DVCS has nothing to do with VCS

I could see myself agreeing to that.

frutiger is correct that the diff algorithm has nothing to do with git itself, in that git can accept pretty arbitrary diff algorithms in the first place for all the commands that take one.

Check out git-diff(1) and --diff-algorithm. --anchored is the one I find the neatest.

DVCS indeed has nothing to do with VCS, it has a lot to do with the data model used by the VCS.

A modern but still centralized VCS like Subversion or Perforce is what you get if you first add networking (CVS) and then atomic commits. Without atomic commits you are pretty much forced to keep a centralized server, and Subversion didn't try to change the server model after adding atomic commits.

DVCS instead is what you get if you start with local revision tracking like RCS, and add atomic commits before networking. Now the network protocol can work at the commit level and is much more amenable to distributed development.

Except you have to run them every time.

Imagine instead of that were available as a sort of materialized view.

> Imagine instead of that were available as a sort of materialized view.

I don’t understand what you mean by this, can you provide some more detail?

Why do we have byte code? Why not run everything in interpreters? Because parsing pure text takes a lot of work. So we store it in an intermediate mode to economize.

Saying just parse it every time is denying that there are very real costs associated with hat decision.

You can store the result of the parse in git if you want to.

It would have to be specific to certain languages, which would, in turn, hinder adoption of new languages to some degree if the git-next took off. So, I'd prefer not to have that be a feature. :)

I think you could write it as the ability to a diff on a binary ast without fussing too much about what the ast represents. Then you merely need to write a parser/serialiser combo for your language to the ast as a repo plugin.

Otherwise it won't just be new languages which suffer, but users of supported languages will suffer when there's an upgrade.

Most language ASTs don't encode unfinished or work-in-progress code very well (the difference in AST shape between missing one `{` and the fixed code can be substantial). You may think it better to always only commit working code, but your source control system is also a backup system if you need to save a work in progress branch to come back to it, and also sometimes a communications system if you want to request a coworker examine your code to help you pinpoint bugs you can't find or review work in progress.

Most language ASTs also don't encode useful-to-the-programmer but useless-to-the-compiler information like comments and whitespace. There's been good progress in that (the Roslyn AST system has some neat features), but in practice an AST is always intended more for the compiler than the user/source writer. This also is reflected often in speed, a lot of languages have a relatively slow AST generation (which would add sometimes very noticeable wall clock time to commit time, depending of course on language and hardware).

Plus, of course, all the usual bits that ASTs are extremely varied among themselves (some are weirder DAG shapes than trees, for instance).

An experiment I ran was to the use the "next step down" from full AST which is your basic tokenizer / syntax highlighter. Those designed to deal well with malformed/unfinished/work-in-progress input, and to do it very quickly. Years back I built a simple example diff tool that can do token-based diffs for any language Python's commonly used syntax highlighter Pygments supports. [1] In my experiments it created some really nice character-based diffs that seemed "smart" like you might want from an AST-like approach but just by doing the dumb thing of aligning diff changes to syntax highlighting token boundaries.

You could even use it/something like it/something based on it today as your diff tool in git if you wanted, with the hardest part configuring it for which language to use for what file. (I never did do that though, partly because the DVCS I experimented with this for didn't have a pluggable diff system like git does, nor did it support character based unidiff as a storage format which the experiment was partly to prove both ideas could be useful.)

[1] https://github.com/WorldMaker/tokdiff

Look into Unison, a language that stores the AST and immutable history of all functions to provide a combination of package manager, IDE, and DVCS. Once you store the AST and all history, some fascinating side effects happen!

https://github.com/unisonweb/unison

I want an editing environment which operates on the AST of my code (obviously it would have to support every language I wanted explicitly to do this), so that files become entirely irrelevant, I never have to worry about formatting differences or where things are or whether that function is in that file or that file. A bit like working in a Smalltalk image.

If that was then extended into the version control system that'd be even better. Oh yes.

But getting a new language into these things would probably be a nightmare.

I don't see any obvious flaws with Git.

The monorepo/polyrepo discussion exists apart from your choice of version control system and has little to do with Git, as far as I can tell

>I don't see any obvious flaws with Git.

Large files and long histories hinder its total dominance in the game and art industries. Because of git's shortcomings polyrepo is a near necessity not simply a stylistic choice. LFS is a bolt on solution that could/should have better support.

> Because of git's shortcomings polyrepo is a near necessity

I'm intrigued by this claim. I've come to the opposite conclusion - that monorepo is near necessity with git because there's no tools for branching/rebasing multiple repos at once.

After using both I can say both have problems. Polyrepos lack tools for working with multiple repos simultaneously, and require more attention to versioning. Monorepos have longer histories, and the large number of objects can hurt performance.

Which one you should use depends on which downsides are less impactful for your use case.

I think he's talking about large files there, which is an issue in GIT btw.

I would use it for storing and syncing libraries of large images for my photography, if it were feasible.

Git stores the diffs in chronological order doesn’t it? I recall reading about someone doing a commercial implementation where the commits are stored in reverse chronological order. I’d been thinking that was github but I’ve never been able to find the article again.

Git's model (which it copied from monotone IIRC) is not diff-based, it's snapshot-based. That is, commits are not stored as a diff to the previous commit, but as the whole state of the tree plus a pointer to the previous commit(s).

As an optimization, when it packs several objects together in a pack file, it can store objects as a delta to other (possibly unrelated) objects; there's a whole set of heuristics used to choose which objects to delta against, like having the same file name. And yes, one of these heuristics does have an effect similar to "reverse chronological order"; see https://github.com/git/git/blob/master/Documentation/technic... for the details.

Git does not store differences between files, but packs (versions of) files (to save disk space) in single .pack files.

> LFS is a bolt on solution that could/should have better support.

Like gitlabs own web ide that seems to ignore LFS rules.

Git is extremely user-unfriendly from the command line.

It would also be nice to have a repo that isn't language-agnostic. It's too easy to track non-semantic changes, like white space.

I don't think that's true. I've been using git's CLI since I started using git a few years ago, and exactly zero of my problems with git could've been solved by a different user interface (be it GUI or a "better" designed CLI). Pretty much all of my problems have been with my lacking understanding of the abstractions that git uses to make all of the powerful things it can do possible.

> I don't think that's true.

You are in the minority. It's such a ubiquitous experience, a running joke in the industry. Saying a tool is useful and powerful, is fine and good. That sentiment has nothing to do with usability.

Isn't that the same thing? If you need to be taught the underlying abstraction to be able to understand the UI you've got a text book example of leaky abstraction.

After I had to unfuck a repository for the n-th time, I trialed a switch to Mercurial, we switched shortly after. I can count on one hand how many times I've had to intervene in the last few years.

> It would also be nice to have a repo that isn't language-agnostic. It's too easy to track non-semantic changes, like white space.

In my opinion this is a problem with programming languages rather than version control. Namely we mix presentation and representation when using text as our source code. In the case of whitespace we have an infinite number of syntactic presentations which all correspond to the same semantic representation. Tooling has been created to try to deal with this such as code formatters which canonicalize the syntactic presentation for other tools. Git itself even has to deal with this because of platform differences, i.e. LF and CRLF.

> Git itself even has to deal with this because of platform differences, i.e. LF and CRLF.

I loathe this about git. It has caused my team members so many problems, like images marked as changed.

git should either be dumb about content or smart, not secretly in between.

I tend to think that it's fine, because I remember what it used to be like... what we have now is the "easy" UI!

And they do occasionally put some new stuff in that helps. Like the recent version which adds new commands to split out the two completely different uses for `git checkout` (making/switching branches and reverting files).

Between CVS and git, Canonical releases an excellent DVCS called Bazaar that I loved. I think we all ended up with git because of GitHub.

There is zero doubt in my mind that it's Github that "made" Git. Without it, it would be just another one of many DVCSes. Git's value isn't inherent, it's all down to network effects.

Git submodules have lots of problems. I wanted it to work like a symbolic link to another repository so that I could develop both at the same time on the same machine. Just like how pip allows me to install a Python package in editable mode. If I make a change in the submodule, the superproject should automatically see it.

Maybe it's time for a version control system to come up and fix a problem no one knew they had.

How about history tracking for file renames doesn't work well? It's hit and miss if git blame --follow works.

Subversion did this better even before Git existed.

Git is completely oblivious to moving code from one file to another. Git blame will never show you the original commit if you just relocated a method to another file. Due to this, refactoring often put additional hurdles into exploring the code history.

Yes -- that's what it means to be a flaw with the entire current generation of version control systems.

"Need a centralized server" wasn't specific to one (pre-DVCS) system, either.

That's true, but it's possible to imagine a system superior to git solving this in someway, in the same way that git solves distributed merges.

cherry picking is a shitshow!

i would love to be able to use a patch-based VC.

You can still generate and apply patches with Git.

I believe GP is referring to Darcs/Pijul-style patches, rather than Git "patches," which are really just diffs. See https://en.wikibooks.org/wiki/Understanding_Darcs/Patch_theo...

yep, nemo got it right. git basically hacks cherry-picks in the same way previous VCS’s hacked branches.

unfortunately there are no patch-theory based VCS’s with a practical level of usability. what git was to monotone, X is to darcs/pijul, where X hasn’t been created yet.

> I don't see any obvious flaws with Git.

Merge conflicts.

Merge conflicts are not so scary, and are an elegant way to handle distributed changes with simultaneous edits to a single file.

If the conflict is huge, rebasing can help you by "playing" the commits from one branch one at a time so the conflicts are smaller / easier to fix.

At a previous job, a team was forced to use checkout-style VCS due to their manager's unfounded fear of merge conflicts; I couldn't go in that office without hearing one developer shout to another: "Hey, can you finish up and check in that file so I can get started on my changes?"

I’ve spent too much time helping others fix bad merges, and I still catch myself making mistakes. There’s a lot of work that could be done for clarity and error avoidance.

Sounds like something that would have been solved by using Darcs instead, honestly.

No VCS will be merge conflict-less without locking files.

I'd rather not lock files.

I'd rather lock files, but be able to unlock them, to be able to coordinate with others and proactively reduce merge conflicts.

The only way to make that simple is to centralize the version control system so that you can have a single arbiter of who has what locked. To add easy locking to Git, you'd have to turn it back into a non-distributed VCS.

I don't think I need to sell the value of DVCS over VCS, but what seems to get lost is that buys you a certain amount of essential complexity, expressed in the CAP theorem and its consequences.

We discussed this deeply on the Fossil forum: https://www.fossil-scm.org/forum/forumpost/2afc32b1ab

We came to no easy answers, because there aren't any. You only get a choice of which problems to accept.

I'm talking about purely advisory locks. Accessing a locked file would let you know who locked it, and if you unlock it, it would just notify them. So it's just a communications mechanism in addition to regular merges.

The alternate method is that locking a file marks you as an interested party to a merge, allowing you to review the correctness of a merge.

This would be purely to avoid changes being lost during merges.

Merge conflicts are a part of any VCS that allows two people to edit the same file at the same time.

However, in some cases (like non-mergeable binary files), it is actually better to have a system that allows one user to take a lock on a file and have exclusive editing abilities. The git protocol has no support for those workflows, and so people end up using a Google Doc or something to track who is modifying what file. Definately a place for improvement.

How do you lock a file in a distributed system? Many people won't be online. Many people won't be on the exact same head.

> Many people won't be on the exact same head.

This consideration is actually irrelevant to locking non-mergeable binary files. It doesn't matter what branch we're on or where the file is located, only that you and I both want to edit the logo. Eventually, either your version must be based on mine, or mine based on yours, since they will be merged.

So it's probably better not to have that file in Git, since it doesn't support the workflow around which Git is based.

It's actually right to store your design documents in Google Docs or a wiki and your code in Git, rather than everything in Git.

It is easy to have one filestore to rule them all and in the darkness bind them, but if you want to do different things with them, you have to do different things with them. I'm not sure that it's possible to unify text file and binary doc based workflows, but it seems we don't have to worry because users automatically use the best tool for the job and it's only hackers who tie themselves in knots trying to make git do everything.

I've worked in teams where developers seemed terrified of merge conflicts, to the point of telling eachother not to edit a particular file which seems absurd. Maybe I don't know any better but they seem like part of life.

When at least one person is making changes that touch large parts of the file, it's very sensible to ask others to not touch it, if you don't want to spend hours merging it manually later.

> Git has a couple obvious flaws, and I expect its successor will be the one that fixes one of them

The successor might very well be git 3.0, though.

> The most obvious (and probably easiest) is the monorepo/polyrepo dichotomy.

Isn't that being addressed in Git with the partial clone functionality?

I’ve read a couple descriptions of the internals of git that don’t disagree with the design of SVN, so I’m not unsure why you can’t theoretically check out a single subtree. Either the documentation is too hand wavy or some implementation details have blocked that possibility.

Are you aware of git-subtree?

Git sub tree is the exact opposite of what I’m talking about.

Also: > The responsibility of not mixing super and sub-project code in commits lies with you.

Is a lie. The responsibility of not mixing code lies with every member of the team. Does the author work alone?

Can you explain more about that? I've worked with both and I feel like monorepo is kinda a pain, but I dont understand where Git fits in to either directly. Seems like it just snapshots files.

It's only a problem at a really large scale. At the scale of Microsoft or Facebook, there are factors that lead to the use of a monorepo being more efficient. At that big of a scale, companies have enough resource to develop internal tooling to deal with the problem (e.g. the use of Mercurial at Facebook).

FWIW, in case of Microsoft at least, it's more a question of product size than company size. Microsoft doesn't use a single monorepo for everything, like Google (so far as I know) does - just look at http://github.com/microsoft/; and that's not even counting all the VSO repos! It uses product-specific monorepos for some large products.

It would be nice if you and everyone else stopped gatekeeping this problem.

We have 600 devs and face these problems. I can assure you we sure as hell dont have the resources spare to reroll git. We're way too busy rerolling everything else.

In this context "monorepo" means a huge repository with many many revisions. Git has several well documented deficiencies in this scenario. See [1] for microsoft's experiences with git.

[1] https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...