That's not a particularly hard problem, all tools which try to solve it (for example, IntelliJ), solve it easily. For example, here's the syntax tree that rust-analyzer builds for this example:
fn foo() { }
fn
fn bar() { }
Syntax tree (whitespace nodes elided for clarity): SOURCE_FILE@[0; 31)
FN_DEF@[0; 12)
FN_KW@[0; 2)
NAME@[3; 6)
IDENT@[3; 6) "foo"
...
FN_DEF@[14; 16)
FN_KW@[14; 16)
err: `expected a name`
FN_DEF@[18; 30)
FN_KW@[18; 20)
NAME@[21; 24)
IDENT@[21; 24) "bar"
...
The problem is, a lot of tools are build with the assumption that the code is correct&complete, and don't try to tackle this problem at all.When llvm was young, one of its design goals was to be useful for live code analysis, to prevent that problem of having two parsers that might not agree.
Sounds like we lost some of that along the way.
It seems like parsing both forwards in the normal way and backwards from the end of the file would be helpful? That is, take the longest suffix that parses correctly and assume any syntax errors are before that for error-reporting purposes.
Most programming languages aren't designed for parsing backwards, though!
For instance, if I have a file that compiles and I start typing in the space between two function bodies, I'm probably implementing a new function. In fact that probability is so high that you should just assume that to be the case, even if the code I've written so far doesn't have balanced nesting yet. All decisions and advice should be based on the speculation that the code will eventually represent a new set of blocks, not a reorganization of the remaining ones.
You can't make the same assumption about if statements inside of a function, or about incomplete multiline comments, but you can assume that anywhere from zero to all of the sibling blocks are about to have their scope changed (but not ancestors).
In short, the gap between how I'd describe a commit and how the diff tool represents it also prevents live code analysis from being able to work as well as it might.