IMHO you're often best off with writing a simple recursive descent parser by hand. For lexing you can often get away with splitting the input string at word boundaries.

For slightly more advanced needs, parser combinator libraries make parsing and lexing quite straightforward. I honestly wouldn't use a parser generator.

Regular expressions are also useful for lexing. I like to work from this example code from the Python standard library docs: https://docs.python.org/3.6/library/re.html#writing-a-tokeni...

Remember, `lex` is just a pile of regular expressions, so it is great for simple parsing in C. There are good reasons it was known as the Swiss Army knife of Unix before scripting took over :-)

IMHO recursive ascent is both easier to write and faster.

I could never understand why you have a parser as a separate thing when this was presented as an up-front decision. Reading this series of posts, where something like a traditional compiler architecture emerges in the same way that normal program architecture emerges via gradual refactoring, made it make a lot more sense to me: https://hokstad.com/compiler .

What skills/languages do you have? It can help tune the suggestions.

Barring that, you can google "compiler tutorial" and just start looking for one you like. One nice thing about compilers is they consist of a lot of stages and while the whole is arguably greater than the sum of its parts, the parts are all pretty darned useful on their own, too.

Javascript is my strongest, and I have enough Python to be comfortable - I had a lot of Perl back in the day but that knowledge is fairly rusty (though Parse::RecDescent was my first exposure to parsers). I did Java for a few years, but I refer to those as my Dark Times and wouldn't want to venture back there. (Experienced Perl dev doing Java is NOT a good time for either the dev or the code)

I can't argue about the broad applicability of compilers, but the main reason I've avoided such tutorials is that I'm looking for something I can get the basics of in hours/days rather than a weeks/months.

For the record neither is experienced Java dev doing Perl.

If you got to the point of having a tree, but found them clumsy to work with, the first thing I'd suggest is to play with an AST explorer. https://astexplorer.net/ is quite good.

Picture the task of what you want to do in your head. Maybe you're writing a linter, and you want to enforce a rule like "never use the 'var' keyword". Write some example code and poke around in the explorer.

Another thing to keep in mind is that once you have a tree, usually you also want to have some tree traversal utilities, so that you can walk over the tree and optionally transform it.

Actually "compilers course" is exactly what you should be looking for imho. Compilers are basically apps that parse input and output something in response to it - which is what this is all about. Lex and yacc are very simple to use once you understand the core principles.

Tbh parser combinators and PEG parsers are the modern way of doing these sorts of things.

PEGs (parsing expression grammars) are probably the by far easiest way currently.