News for hackers. Stuff that matters.
Re:How Microsoft rewrote its C# compiler in C# and made it o... by Sharlin on Tuesday February 11, 2020 @04:39PM (#22303179)
Pardon me if I'm missing something, but doesn't just about any AST implementation allow recovering the source text up to whitespace – or indeed including whitespace, given that most real-world parsers have to retain lexical information in order to support reasonable diagnostic messages anyway.
Re:How Microsoft rewrote its C# compiler in C# and made it o... by Rusky on Wednesday February 12, 2020 @11:41AM (#22310293)
Many (if not most) ASTs, especially in batch compilers, do not preserve comments, whitespace, or even all of the actual syntax. Even the diagnostics information you're referring to are insufficient to reproduce the whitespace.

Further, what Roslyn does is quite interesting even on top of preserving all of that. It does so in a way that lets it incrementally re-parse parts of the program without touching the rest- including position information for the rest of the file after the change, which has all shifted to match.

None of this was standard practice anywhere when Roslyn was designed, though it's starting to be picked up in other projects since.

Re:How Microsoft rewrote its C# compiler in C# and made it o... by Karliss on Tuesday February 11, 2020 @05:47PM (#22303817)
There is reason AST is called abstract syntax tree not just syntax tree. Many syntax details like parentheses or choice between alternative syntax forms are there only to avoid syntactic ambiguity, improve readability or historical reasons and don't affect the semantics or even error messages. It's not surprising for compiler authors to choose discarding some of the unnecessary information as early as possible to save memory.
Re:How Microsoft rewrote its C# compiler in C# and made it o... by ygra on Tuesday February 11, 2020 @06:19PM (#22304047)
Notable a lot of JavaScript AST libraries at least a while back omitted comments and whitespace, which is fine for writing a minifier, but horrible when trying to implement source code transformations.

What I really like about Roslyn's AST (and I guess TypeScript's looks about the same) is that all those details are put into Trivia; ignoring those is just passing one flag to the visitor; you don't have to be aware of how many different types of AST nodes there are that might be ignorable. It also means that each and every whitespace and comment belongs to a syntax token instead of being other nodes interleaved with the normal nodes. This helps keeping comments where they belong even after re-arranging code in the AST – something that, e.g. ReSharper is atrociously bad at.

Re:How Microsoft rewrote its C# compiler in C# and made it o... by lonelappde on Tuesday February 11, 2020 @07:05PM (#22304363)
It's called abstract syntax tree because the concrete syntax is not a tree, it's an list of characters. Not because it makes semantic-invariant transformations.
Re:How Microsoft rewrote its C# compiler in C# and made it o... by yatac42 on Wednesday February 12, 2020 @06:29AM (#22307939)
> It's called abstract syntax tree because the concrete syntax is not a tree, it's an list of characters.

It's called abstract syntax tree to distinguish it from parse trees (a.k.a. concrete syntax trees).

Re:How Microsoft rewrote its C# compiler in C# and made it o... by cf on Tuesday February 11, 2020 @05:56PM (#22303865)
Many times things like comments are dropped and names are disambiguated. Printing the tree back out would yield an equivalent program but the source code would look way different.
Re:How Microsoft rewrote its C# compiler in C# and made it o... by _hardwaregeek on Tuesday February 11, 2020 @04:51PM (#22303306)
I'm not sure to be honest. I suspect most parse trees preserve most of the info but not up to bijection. For instance, Ruby has ways of taking the Ripper output and generating source text but there's no guarantee that this output will be precisely the original source text. After all, the parser would need to distinguish between spaces and tabs and store them as tokens to reproduce the text fully.
Re:How Microsoft rewrote its C# compiler in C# and made it o... by fierarul on Tuesday February 11, 2020 @05:07PM (#22303444)
> doesn't just about any AST implementation allow recovering the source text up to whitespace

That's correct. Especially for parsers used in IDEs you most definitely care about exact locations of everything, including whitespace.