Why don’t we just check ASTs in to git instead of text files ?

I think this is the key question, and doing this will enable numerous interface improvements for people working with software (Eg: separation of content and presents means that everybody can use their favorite view — three-space-tabs, extra-long lines, whatever).

The challenge is that version control on text files (list of lines) is fairly straight forward and language agnostic, while OTOH the VCS would have to run after the language parser processes on the text file, and then be able to interface with the AST generated by the parser (for multiple languages). I presume this would become easy if different languages have the same parsing setup, or we have language/project-specific VCS tools. While the answers are not obvious (to me, at least) this definitely seems like the right direction to move in, instead of getting hung up on plain text and formatting issues.

Storing an AST instead of text might be a way to eliminate parsing ambiguity of angle brackets for generic arguments (as opposed to comparisons). It might still be visually ambiguous, but I don't think it's a problem in practice (if the display forces whitespace around comparisons)?

There are some cases where formatters like Prettier don't have one canonical representation for an AST, in particular behavior around line breaks (multiple empty lines are allowed, a line break after an "{" opening brace in an object literal will force it to span multiple lines, etc)

But yeah, otherwise everyone could just have their own preferred formatter/settings and we could commit whatever representation is most optimal for diffing to version control (e.x. multi-line object literals, trailing commas in arrays/args, etc).

You could use something more compact like a binary AST but for compatibility with "legacy" tools that deal with text I think it would be better to use a more verbose text-based representation.

I believe unison does something along those lines:

https://www.unisonweb.org/

I believe that's called "lisp".

Which lisps don't use text files for source code?

I kind of like the ability to have comments, though. The tools I've worked with have issues round-tripping them though ASTs.

I have thought about this a bit in the past, and I think the answer is that the AST for a given blob of text can be different depending on which version of the language you interpret it with. consider 'print "foo"' in python 2.7:

    Python 2.7.15 (default, Feb 14 2019, 16:26:56)
    [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import ast
    >>> ast.dump(ast.parse('print "foo"'))
    "Module(body=[Print(dest=None, values=[Str(s='foo')], nl=True)])"

but in python 3.7:

    Python 3.7.2 (default, Feb 14 2019, 16:12:42)
    [Clang 10.0.0 (clang-1000.10.44.4)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import ast
    >>> ast.parse('print "foo"')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/philsnow/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 35, in parse
        return compile(source, filename, mode, PyCF_ONLY_AST)
      File "<unknown>", line 1
        print "foo"
                  ^
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("foo")?

nothing changed in the text, but the AST is different. What should get committed to this hypothetical "git-ast"?

The answer probably lies in creating a different abstraction , different building blocks. In git we have bytes -> blobs -> trees -> commits, where the bytes of a given file get annotated with a file name (and mode bits etc etc, gross posix things) by the tree object.

In git-ast, maybe the abstraction is bytes -> lexemes (which annotates the bytes with a language version, and thus lexer version) -> tree? I haven't gone down the thought experiment that far.

I suspect you're probably right, but that example isn't the greatest because Python 3 isn't source compatible with Python 2. The source code text itself isn't even backwards compatible in that case.

That was maybe not a great example, pick any language change where a new keyword was introduced that wasn't previously a keyword or reserved word.

I tried to make an example with `with`, because that was introduced between 2.4 and 2.5 IIRC, and 2.4 and 2.5 aren't nearly as far apart as 2.* and 3.*, but I didn't find an easy way to install a 2.4 python (it didn't compile in pyenv on my mac, and I looked at a couple docker images before giving up).

But you're describing a benefit of storing an AST. Instead of the code suddenly changing meaning, you have an AST that says what the meaning should be.

The worst case is that you export it with the old version of the converter, and then the code changes meaning. You have silent corruption. But if you stored raw text this would always happen.

The good case is that you export it with the new version of the converter. It will notice that a variable name matches a keyword, and alert you to fix it.

Since even minor releases can introduce new syntax that previously failed to parse, this happens more than just on major version numbers. For example: async/await, formatting string literals, underscores in numeric literals, the list goes on and on. So you'll have to upgrade your formatter. And when you upgrade it, it may start formatting something differently than it previously did, even if that chunk had no new syntax.

Use one that won't change how it formats on a whim?

Because that requires either compiler support, or a plug-in to be written for each language and language version. But, I guess you can always fall back to text files in the worst case.

Sounds like a great idea, or binary representation of lexer tokens, the AST is larger in size than the lexer tokens and the AST can be easily generated from the tokens

This is basically solved by pinning the formatter to a specific version in your project's dependencies, and enforcing its use in continuous integration.

Of course that doesn't help if you're setting up the formatter for the first time, or want to upgrade to a new version.

Some tools like GitHub's diff do try to deemphasize whitespace but it's pretty limited.

In easy steps (Python/Git specific):

1. Use a tracked directory for Git hooks: `git config --local core.hooksPath 'git-hooks'`.

2. Create format.bash and lint.bash.

2.1. Run `black` and `isort` in format.bash.

2.2. Run `flake8`, `mypy`, `black` and `isort` in lint.bash.

2.3. Run lint.bash in the pre-commit hook.

2.4. By default, lint/format only files which are different from origin/master to save time: `git diff --diff-filter=d --name-only -z origin/master "[asterisk].py"`.

3. In CI, run lint.bash with `--all` or some other keyword to verify formatting on every file: `git ls-files -z -- "[asterisk].py"`.

After more than a year with this setup I absolutely adore it.

Even better, IMHO, is just setting your editor to format on save. That way you can just write ugly code and hit save to get it formatted.

I've found git hooks to be more trouble than they're worth.

But as long as it's enforced in CI you can leave it up to individual team members how they want to do it.

Append “?w=1” to the end of a Github PR URL, it’s like passing “-w” to git. (Not that that addresses all of your comment, I just wanted to share the tip. Agreed that smarter diffing would be really nice.)

git diff has a --diff-algorithm parameter that has several options to choose from. I don't know whether Github supports that via url parameters though.

The frustration is real in a PR situation when you have your editor set to run a formatter on save (same thing with stripping whitespace) but if you have enough control of the project and want to switch to using a formatter then git can help you there.

You can do a format run over the whole codebase then ignore the commit in blame.

https://www.moxio.com/blog/43/ignoring-bulk-change-commits-w...

Hopefully you're already aware but just in case: many tools (including git and github) have an option somewhere to hide whitespace changes. In some contexts this can help a lot, though it certainly doesn't get us all the way there.