I'm a bit confused by the benchmark. I looked through the benchmark code and, without really understanding what I'm looking at, the result of the nom parser looks much closer to the "pest (custom AST)" benchmark than the "pest" benchmark. The description above the image, however, primarily compares the "pest" result.

Am I misunderstanding something or is the comparison not fair?

I've added a clarification. The point of the benchmark is not to compete with other projects, it's merely there to put the parsing speed (and not necessarily the processing that comes after it) in a representative window of performance.

This is a neat project! I might take a look at this if/when I get started on my dream language. :p

One constructive criticism, which may be only an unpopular opinion, but I find creative operator overloading unnecessarily hard to read. And I don't think my opinion is entirely subjective since, by definition, you have to learn new semantics whereas a good function name would make the meaning obvious. Maybe this syntax would be familiar to people who are already familiar with formal grammar notations?

I agree with your point. I tried my best not to sway too far off of some of syntax I've seen in other PEG projects, while also keeping it compatible with Rust's standard macros. While pest does have this constraint any longer with the new beta, changing the grammar too much would have been a nuisance for anyone who had grammars written in the older versions.

Understood. Thanks for taking the time to explain!

What type of parser does this produce?

For now, it produces a recursive descent parser. Packrat parsing is an open-question since I'm afraid that adding a memoization layer all throughout the parser will lead to a consistent general slowdown.

Interesting. I'll try and find some time to read the source - am interested to see how this is implemented.

A good place to start would be in the manually written example. [1] My current plan is to try and limit the the use of memoization such that it still guarantees linear parsing, but it doesn't memoize unless necessary.

[1]: https://github.com/pest-parser/pest/blob/master/pest/example...

I appreciate the link - I'm teaching myself about parsers right now and like the rest of HN I like Rust programs ;)

"Parsing expression grammars (PEGs) are an alternative to context free grammars for formally specifying syntax" [1]

Basically, you have tools for context free grammars such as GNU Bison or Yacc. This is an alternative, but with PEG.

For more in-depth info, there's the original PEG paper. [2]

[1]: http://bford.info/packrat [2]: http://bford.info/pub/lang/peg

That's the grammar, not the parser resulting from the grammar.

The README doesn't say, but most PEGs in the wild (AFAICT) compile down to / are implemented by packrat parsers.

With PEGs (like Pest), it's probably going to be a packrat parser (which would also make sense given the project's name).

Thank you for the actual answer. Funny how some people are so excited to give an RTFM answer that they forget to read the question.

E: turns out this is wrong anyway.

I remember playing with PEGjs back in the day, making a toy implementation of JS in JS. It had terrible Unicode support though. To build a token out of valid Unicode non-whitespace characters took pages and pages of range definitions. Is that any better here?

Rust by default uses Unicode for strings. I would be surprised if this had problems with them.

The difficulty wasn't in the engine. JS itself is just as good with accepting Unicode characters as Rust, both in code and strings. The problem was with the PEG implementation. Without some sort of shorthand for Unicode families, defining programming languages that are valid in multiple real-world languages becomes a serious burden.

This is one of the things Perl 6 does really well,).

Been using this for a couple weeks now and dig it. The DSL feels familiar and the speed is yummy. I'd been using LALRPOP for some months; I dig it too, but just couldn't get the speed I was after from it.

Shouldn't the title better by "Pest - Fast, modern parser _in_ Rust"? The current title made me think that this would be a Rust AST parser, similar to syntex.

The confusion here is because Pest is not, in fact, a parser. It's a parser generator: you can use it to generate parsers (maybe even a parser for Rust), but it is not in itself a parser.

A better title would be "Pest - Fast, modern parser generator for Rust".

One cool thing is that rather than the parser generation being a standalone tool, it's written using Rust's macro system. Normal parser generators use some sort of command line tool to generate say C code which can then be used like any other library. Here, there's no need for generated code, because the Rust macro system generates it on the fly.

Exactly what I came into this thread expecting. Was curious why when it looks like there's already great IDE support for Rust with existing tools.

There's actually a parser for rust called syn, and it's very useful for procedural macros, as the current API exposes a stream of tokens, not a full AST.

To elaborate a bit on that, exposing an AST directly means that you're tied to that version of the AST; this was considered really hard for compatibility over versions. Token streams are much easier to define an interface for, and more flexible: you're not just stuck with whatever AST representation we'd have given you.

For example: https://doc.rust-lang.org/book/first-edition/procedural-macr...

I agree. I guess I was trying hard to suggest the fact that it can be used _with_ Rust and not the fact that it is written _in_ Rust.

OK we've attempted to make a more accurate title above.

Yes this is no doubt the current implication. I was expecting something like racer.