I think writing a parser by hand should be reserved for when

1. You have users.

2. You can demonstrate a marked improvement in performance and/or error reporting with the hand-written parser.

Otherwise, deriving a parser from a grammar should absolutely be the default. Writing parsers by hand is error-prone, causes security vulnerabilities[0], and is very tedious work.

[0]: http://langsec.org/

Having written production-level compilers, my advice: use ANTLR for as long as humanly possible. It can remain adequate far, far longer than you think, and will save you months/years of stress.

When you outgrow it, you'll know - and by then, hopefully you'll have funding/users/contributors/etc to support the dev effort of rolling your own parser.

Our compiler stack is in Haskell so we initially started with Parsec (parser combinator library) and then eventually switched to Happy (parser generator that Haskell uses itself) since at first we didn't have a formal grammar defined so that made us write a lot of code for every little addition to the language (it's a simple declarative DSL, https://github.com/wasp-lang/wasp).

The problem we're now facing with Happy is that it doesn't really have support for error recovery. We're trying to implement LSP so recovery is important in order to have proper syntax highlighting and error reporting in an IDE.

Wondering how well ANTLR covers this, I was quite surprised to find out that Happy has so little support for it.

IIRC Happy (and Alex, the accompanying lexer generator) are pretty much straight ports of lex/yacc to Haskell; they predate some of the conveniences of newer parser generators like ANTLR.

I would have the opposite advice. Derived parsers have shit errors. Which is arguebly the most important feature of a parser besides being able to parse. And writing parsers by hand is trivial anyway. In a lot of cases if you squint a little you can see the BNF.

3. Your language has very little/no syntax.

A parser for a Forth is a one-liner or a short loop in most languages.

4. You want to learn how to implement a parser by hand.

Just write recursive descent, except for the infix operator precedence, and then use a tiny stack that you either push to or pop from (shift/reduce, ala LR) based on the relative precedence.

Pretty much all compilers that have fast parsers that also give good error messages are hand-written recursive descent.

It's a fine book and you should not tell people to read it.

It belongs in a first compilers course where you learn about how scanners and parsers work, not just "configure this then call `yylex`". So yeah, it nerds out about finite state machines for a little too long for most audiences. And you know your audience. Your audience wants to write a compiler today, so you are correct not to recommend this book.

> Actual compilers tend to handwrite their parsers rather than using theory; parser generators aren't good at recovery and error messages.

The linked article recommends using a parser generator. However in my experience I agree with you, parser generators aren't great in practice, which makes me hesitant to read the rest of the post.

The time investment of writing a good hand-written parser is only worth it if you have users whose workflow would be improved by performance/error reporting gains once you have exhausted what a grammar-derived parser can provide.

For Clang and GCC is it absolutely a no brainer to use a hand-written parser.

If you're trying to get to an MVP compiler, or you're building a bootstrapping compiler, using a grammar-derived parser to get to the finish line faster is far more sensible.

Using parser generators doesn't make a lot of sense to me.

It's far easier to handwrite parsers.

Unless you're a midsized company with the resources and who doesn't mind bad error messages I would not recommend anyone use parser generators.

What could be easier than writing a grammar in EBNF-ish notation? This gives you the added benefit that the grammar is explicitly defined, and not implicit in the parser code.

> This gives you the added benefit that the grammar is explicitly defined

Especially if you're still refining the syntax: it is so comparatively easy to modify a grammar vs parser.

> What could be easier than writing a grammar in EBNF-ish notation?

Not learning a new language/build system/coding paradigm. :)

ANTLR took me an afternoon, before I could use it for a simple language, including adding it to my build tool.

> It's far easier to handwrite parsers.

It's also far easier to make buggy handwritten parsers. I don't have much experience, but in the project that I've seen, where a handwritten parser was used, the issue tracker was full of parser bugs.

I tend to write lots of tests at each level -- lexer, parser, AST, etc. -- to cover the different BNF grammar statements. I've done this with an XQuery and XPath parser I wrote for IntelliJ.

You can cover the different lexical variations in the lexical tests, and ensure that the lexer is reporting the correct things in the lexer tests. For these, I generally ensure I cover each token in a given BNF grammar statement, even if that means repeating checks for a given keyword, as it keeps the tests self-contained and easier to verify.

You can cover the different parser/grammar variations (different branches/paths in the BNF, missing tokens for error reporting and recovery, etc.) in the parser tests. These can make use of serializing the tree to a file, so your input is an example program and the output is the parsed AST tree.

The AST tests can then test the actual data model your AST provides, such as access to the value of a literal, or information about a type.

Higher-level tests such as constant folding can then be done using the AST API, or things like serializing the AST to a textual representation.

It's also far easier to debug handwritten parsers.

a parser generator can be a very good first pass at determining whether your language is properly formed

What do you mean by properly formed?

Probably that it doesn't have ambiguities

Thanks, that’s a better way of putting it.

Is that an important property? If so, why do you (*whoever) think so?

As always, it depends on what your goal is.

An unambiguous parser is easier to write tests for. It leads to fewer subtle "gotchas" where something syntax checks and compiles but is interpreted by the compiler as something very different than what the programmer intended. It's easier for third parties to write their own parsers and tooling for the language if they can target a formal grammar as opposed to the behavior of your Turing complete code.

But sometimes you don't care about any of those things. Especially if you're talking about a language with a small surface area. Or if you have to deal with legacy stuff (for example, parsing something like C++ that is heavily context sensitive, in which case trying to shoehorn things into a formal CFG so you can feed it to a generator might actually make things much harder than they have to be).

Not always, and that’s a great question! But I can help you catch things that might not be obvious when your first create the grammar, even such simple things as ambiguous nested if statements or errors in string formatting DSL syntax.

* it can, not I can

It takes 1 or 2 hours to make a decent pratt parser for even somewhat unusual syntax.

Regarding parsing techniques: https://notes.eatonphil.com/parser-generators-vs-handwritten...

I agree it is a bad book, but you don't seem to be explaining why.

The article says exactly what I'd say. It spends far too much time explaining the frontend (parsing theory) which isn't the difficult or important part of the compiler, and it somehow makes it sound scary and mathematical.

The rest isn't covered enough and doesn't have enough practical real-world advice.

As someone who has had to clean up (with backwards compatibility) after people who YOLO'd custom grammars rather than actually taking the time to respect the mathematics of parsing and design ambiguity out up front...

I cannot disagree with you more.

I mean, I think you should read a book on parsing theory, just not that one. I like "Parsing Techniques: A Practical Guide" but there might be a newer better one.

But clang and gcc both have handwritten recursive-descent parsers for C/C++. (C++ parsing of course is undecidable, or at least used to be.)

C++ parsing being undecidable, and C having the Most Vexing Parse, are the classic arguments for developing a grammar using a parser generator, and only when the grammar is mature, write a handrolled implementation using the grammar as a reference.

I happen to think generating useful software from declarative grammar specifications is, hmm. Not limited to what such frameworks have been capable of to date. Although if you've only worked with yacc/bison, and never ANTLR, you may be underestimating the quality of a generated parser with respect to error recovery and messages. The state of the art has advanced, I don't think anyone would claim parity with artisanal handcrafted softwares, but we can get a lot more than "barf: shift-reduce bunchanumbers" these days.

> and C having the Most Vexing Parse

C++ too. Just to be pedantic.

I can echo this sentiment. Last year I took a course for my master's where we implemented a compiler end to end. The dragon book helped for most of it but when it came time to implement the assembler we were unable to get it completely correct and the book was very scant on any information about how to best manage going from virtual to physical registers, handle storing information on the stack, etc. Granted some of that is platform specific but the difference in material from frontend to backend was stark.

Parsing is also about 1% of the effort for a mature frontend.

Books do not cover how to actually structure a compiler and write sensible passes etc.

Frontend is very important if you want your language to have good IDE tooling (which most languages do not have)