Taking a compiler theory class did wonders for me as a programmer, even though I am in no way capable of actually writing a compiler without a lot of learning curve.

The real falling-down place for most programmers, I think, is that they have no clue how parsing works. The widespread prevalence of XML/JSON, with convenient libraries, has made data stream construction/parsing a thing of the past (which is good, because most of the data stream protocols I saw were awful, filled with shift/reduce and reduce/reduce ambiguities, basically just giant hand-rolled case statements).

Yeah, and most of the tooling that isn't written in C (Lex, YACC, and MPC come to mind) has documentation that assumes you already understand the API. I mean, what?

On top of that there are a lot of other practical problems with those tools. It's hard to learn to use them to do simple things, and even harder to use them to do moderately hard things.

For instance in a lot of cases what you really want is to convert a grammar to some reasonable abstract syntax tree and I've found often you have to write a lot of code into those frameworks to do that, whereas you ought to be able to write something declarative that generates the AST objects and the translation code.

Another problem is a lack of composability. Particularly in DSL creation it would be desirable to "cut and paste" grammars into each other; it's very possible to embed SQL into a language like C, Java, or such -- you ought to be able to embed one parser in another but generally it is not easy.

Also there is the problem of error message generation, which is bad enough when you do are doing things that are "easy with yacc" but when you are doing hardcore transpilation of some kind, the error messages are often completely incomprehensible. This is a big factor in any kind of "rules engine" built around an existing language; error handling is disastrously bad in Drools, for instance -- you might think homoiconicity would improve matters with Clara in Clojure but it really doesn't.

...And unless you read the dragon book, you won't know how to write a recursive-descent parser. As for nested grammar, I think MPC supports that the best of the three, but even then, the support probably isn't the best. I haven't really checked though.

If you want this kind of composability, use PEG-based tools.

Same for the error messages and error recovery.

Wait a sec. Since MPC is a Parser Combinator library, wouldn't it have the composability?

Yes, you're right - did not even notice MPC in the list, was commenting on Yacc and alike.

Yeah. I wasn't commenting on specifically YACC and Lex, just compiler toolkits with good documentations. And MPC has some of the best documentation out there. Also, Build Your Own Lisp is amazing.

This is an extremely common problem with documentation, it's not limited to these kinds of tools.

The fundamental problem is: you understand how something works, and you are trying to communicate with someone who doesn't. In a conversation, your partner can ask clarifying questions when you accidentally say something that's not clear, but in text, you're separated by time and space. So this can't happen.

It can happen.

Ever hear of a wiki, or an issue tracking system?

Literate programming systems have failed so far because a "code-document" bundle is not the way.

The way is to attach whatever documentation exists to a scaffolding that includes the code. For instance, stackoverflow contains a large amount of documentation which is poorly organized and is only findable (sorta kinda) because Google is (still) a good search engine.

  > Ever hear of a wiki, or an issue tracking system?

... yes.

I didn't say that it was impossible, just that it's a very common issue, more broad than just compiler tooling.

Well, yeah. but compiler tooling has it far worse than most things. And I was reading through the docs to some compiler tooling written in scheme. Shudders. It has a Lex/YACC like grammar DSL, but it doesn't explain the syntax, just provides a fairly bad example. And this was (chibi parse), which was written by Alex Shinn, who usually does pretty GOOD documentation (irregex in particular is excellent, assuming you already know regex).

Correct. I am tirelessly ranting that the major culprit here is the infamous "Dragon Book". It somehow managed to turn an easy and fun subject into something intimidating and thoroughly boring.