No, the Ruby grammar is actually an outlier from what I've seen; it has one of the largest/most complex external scanners: https://github.com/tree-sitter/tree-sitter-ruby/blob/master/...

Precisely because the language is complicated and less amenable to LR parsing.

Not a ruby developer here: that sounds terrifying! Does it make it harder to have a proper mental model of the language (note: not the libraries) or is this mainly because of flexibility (too many ways to skin one cat)?

I don't write Ruby regularly either, but I wouldn't say that syntactic complexity, is necessarily equivalent to semantic complexity. And the syntax is the only part that's relevant to Tree-sitter: it's not an interpreter/compiler.

Note also that (as I alluded to above) the parsing technique that Tree-sitter uses, "LR parsing", makes some things more difficult to parse than they'd be with another kind of parser. This is a deliberate trade-off, because LR parsing makes certain features of Tree-sitter, like fast re-parsing in response to input changes, much much easier.

So, a syntactic tree is a list of elements, grouped by their ordering, which are to be parsed from their arguments, as they appeared in the input. Or a grammar tree, which is a set of elements. There's many things we can do to make Tree-sitter simpler to read and write. Perhaps, like in Perl, there are syntactic categories of types that make it much easier to find things like nodes in a tree, since they're the ones that come in the input. Or I'd be willing to say that maybe, like in Haskell, certain aspects of the language, are syntactic categories, like the parser. So some things that might not be obvious in code, like what the syntax for a class of names is, might be obvious in theory, too. Or, at least they might be obvious in a particular way. Or some aspects of the compiler are really special, and we can infer those in terms of what the compiler does. Or, of course, we can do all these other things, too. We can rewrite the parser, or the compiler, to try to do more or less anything that the parser does. Or maybe we can make Tree-sitter a lot simpler in general. Which I think is probably what you've been thinking about.

Flexibility. “Too many” is debatable: most organizations wind up settling on a subset of the idioms that Ruby provides, and some of the more esoteric constructs see infrequent use anywhere.

There has been, however, discussion about the need to clean up some of the lesser-used language feature, but obviously doing so carries risks.

> Not a ruby developer here: that sounds terrifying! Does it make it harder to have a proper mental model of the language

It is a little terrifying in the sense that I'd not want to write language level tools (eg: syntax highlighter).

But if you have scheme on one end and natural language on the other, ruby leans à bit towards natural language - but in a good way. In some ways ruby isn't that different from Smalltalk - but it has a lot (sometimes I think too many, sometimes not) conveniences.

Parantheses and brackets are largely optional "where it makes sense". Conditionals support postfix, eg these are equivalent:

  if should_send?() 
    send_mail({to: '[email protected]'}) 
  end

  send_mail to: '[email protected]' if should_send?

My mental model of Ruby is one the simplest of any of the languages I've worked with, but it's also the hardest to put into any words. JS actually does beat it out, and then Scala and Python come after.

Everything is kind-of-but-not-really an object, a reference, and a function, all at the same time - which sounds complicated but in my head... turns out to be pretty simple. Everything's just kind of different flavors of the same thing. `attr_accessor` is a good place to see this in action.

The flexibility comes more from the variety of available core language options (procs, blocks, and lambdas) and core libraries (map/each/collect, for example), not from a variety of underlying concepts.

It's mostly to work less surprising to the programmer, AFAIR. Probably the most complexity is from having to differentiate local variables and methods depending if the symbol had an assignment before in the scope.

This is more a function of Ruby than of tree-sitter. The tree-sitter grammars for other languages are hopefully less inscrutable. For Ruby, we basically just ported whitequark's parser [1] over to tree-sitter's grammar DSL and scanner API.

[1] https://github.com/whitequark/parser

I didn't mean the tree-sitter grammar was not understandable - it's very understandable - I just can't work out how to managed to find such a concise way to express grammars. Even compared to Whitequark it's 1/3 the size. What's the unique thing you do that makes it so concise?

It also seems somehow to be completely declarative? How have you managed to transform Ruby parsing to be context-free? For example where's the set of what's currently a local variable so you can distinguish from method calls?

Ahh my mistake! :-)

To be fair, we're cheating a little bit because the Ruby grammar relies so heavily on an external scannar, which is just under 1,000 lines of C++: https://github.com/tree-sitter/tree-sitter-ruby/blob/master/...

But for example how do you parse the difference between `x = 14; x` and `y = 14; x`? In the latter case `x` is a method call, and in the former it's a local variable read. I can't see where the parser maintains a set of local variables and where it queries this set. Is it somehow done declaratively? If so that's a huge achievement I don't think that's really been done before in a parser generator.

I really want to try tree-sitter for using in an actual Ruby implementation because it's so beautiful!

[EDITED to make the example actually line up with OP's test]

There's no symbol table in the parser, so at parse time, we don't distinguish those cases:

  $ cat test.rb
  module Test
    def test1
      x = 14; x
    end

    def test2
      y = 14; x
    end
  end
  $ tree-sitter parse test.rb
  (program [0, 0] - [9, 0]
    (module [0, 0] - [8, 3]
      name: (constant [0, 7] - [0, 11])
      (method [1, 2] - [3, 5]
        name: (identifier [1, 6] - [1, 11])
        (assignment [2, 4] - [2, 10]
          left: (identifier [2, 4] - [2, 5])
          right: (integer [2, 8] - [2, 10]))
        (identifier [2, 12] - [2, 13]))
      (method [5, 2] - [7, 5]
        name: (identifier [5, 6] - [5, 11])
        (assignment [6, 4] - [6, 10]
          left: (identifier [6, 4] - [6, 5])
          right: (integer [6, 8] - [6, 10]))
        (identifier [6, 12] - [6, 13]))))

In both cases the bit after the semicolon just parses as (identifier).

For some use cases (e.g. syntax highlighting, depending on your colorization rules) it doesn't matter, and so we don't want to pay the cost. If it does matter (like in an actual implementation), then you'd have to implement this yourself and drive it by the parse tree you get from tree-sitter.

Right you could just have a phase to fix-it-up after parsing. Much better than trying to shoe-horn an imperative action into a nice more-pure parser. Great idea!

So what's cool is that while we don't handle that during parsing, you can use another set of tree-sitter features to do tree queries to achieve this. Here's the query for detecting Ruby locals: https://github.com/tree-sitter/tree-sitter-ruby/blob/32cd5a0... and here's some better documentation for how the query language works: https://tree-sitter.github.io/tree-sitter/syntax-highlightin....

The code is obviously much simpler than its syntax - most importantly, its syntactical simplicity makes it way easier to deal with. So when you write the code to parse it you don't have to try to parse it in one fell swoop like you do in Whitequark.

So you can't read anything from a method call! I can make it so, if you're doing a class method (of any kind) you have to invoke the constructor, as described in "What is a method?" There's also a few new techniques like "new_class_method", which requires creating an object (of some kind) for that class... but what about that? It's not "I've just fixed Tree-sitter's problem"; it's that Tree-sitter hasn't yet resolved the problem yet - there are other parsing problems besides Tree-sitter in Ruby itself like those of classes (and classes are not part of Tree-sitter) and things that are known as "type-traits" and so on - so as it's not quite enough it can be done by other things. The reason for using LR grammar is that when it comes to this - what do I want from that grammar?

The point I'm making here is that LR doesn't give a reason for what you're doing. As a programmer you are trying to write code that is portable because - if it works in a domain you don't understand (such as Ruby) - then you don't know what you're doing is wrong. There can be a domain (as in any language) that's a lot more complex than this - but since we've got that, how can I be sure it won't mess up the code I'm writing?

> The code is obviously much simpler than its syntax

What code? The parser? How can it be simpler than its syntax? It has syntax and semantics, which is strictly more than the syntax.

> The point I'm making here is that LR doesn't give a reason for what you're doing.

What do you mean 'what you're doing'?

Hey thanks! I'm one of the primary developers of this grammar along with @maxbrunsfeld. It was the driving force for supporting an external scanner and while there are still some Ruby edges cases, I'm pretty happy with how it came out. I will say we spent a lot of time on this and I read both the bison Ruby grammar and whitequark's ruby parser (which is excellent) in great detail to understand how to deal with certain parts of the language.

One thing I love about tree-sitter is how both the grammar and the resulting ASTs are so readable. I can come back to this project after months of not contributing and pick up right where I left off.

bison should be compared to https://github.com/tree-sitter/tree-sitter-ruby/blob/master/... probably?

No the JSON file there is generated (I believe?) from the JavaScript I linked, while the Bison file is hand-written.

With tree-sitter you're hand-writing a 1k file. With Bison you're hand-writing a 13k file.

@chrisseaton you are correct, the JSON file is generated. The handwritten parts are:

- https://github.com/tree-sitter/tree-sitter-ruby/blob/32cd5a0... - https://github.com/tree-sitter/tree-sitter-ruby/blob/32cd5a0...

So about 2k loc.

The trickiest (and most verbose) parts of the external scanner have to do with heredocs and the various ways to declare literals (strings, symbols, regexes, etc).