Tree-sitter – a new parsing system for programming tools [video]

Re:Tree-sitter – a new parsing system for programming tools ... by maxbrunsfeld on Tuesday October 16, 2018 @09:45PM (#18235074)

Yeah, the grammar author does fully control the division between the parser and lexer: every literal (string or regex) in the grammar corresponds to a token. There's also a `token()` function that you can use to specify that an arbitrary rule should be handled by the lexer as a single token.

In most cases, you don't have to think about it; the obvious way to write something is the right way. There are cases where its helpful to have a mental model of how lexing works.

[ Parent | on HN ]

Re:Tree-sitter – a new parsing system for programming tools ... by mncharity on Tuesday October 16, 2018 @10:34PM (#18235645)

> grammar author does fully control the division between the parser and lexer

Nifty. So explicit GLR forks at rule level (`conflicts:`), non-forking token "conflict" resolution[1], no token regex backtracking pressure across tokens, and token resolution at a single code position can(?) differ across conflicting rules? I'm uncertain on that last bit.

> In most cases, you don't have to think about it

Well yes, but, some of us crave much more syntactically flexible languages. :)

[1] https://tree-sitter.github.io/tree-sitter/creating-parsers#c...

[ Parent | on HN ]