News for hackers. Stuff that matters.
Re:Adventures in the land of substrings and RegExps by evmar on Saturday December 03, 2016 @10:43AM (#13096078)
I worked on a project where lexing speed was of concern. Because it was C++ I didn't have the easy option of reaching for regexes so I instead wrote a hand-rolled lexer as suggested in the post.

But later I found a tool, re2c, where you write regular expressions in your source code and it expands them out to a fast parser using every last trick (gotos and lookup tables, see an example of the generated code here[1]). The result had the delightful and rare property where the final code was both higher-level (regexes are a succinct way of expressing lexing) and faster than the lower-level hand-rolled code.

https://github.com/ninja-build/ninja/blob/3082aa69b7be2a2a06...

Re:Adventures in the land of substrings and RegExps by mraleph on Sunday December 04, 2016 @05:30AM (#13099508)
I guess I should have been more clear in my advice about not using regular expressions: if you have a tool that allows glueing multiple regular expressions together and erasing the cost of the abstraction (e.g. no allocation of temporary match result objects, substrings, etc) - then certainly that would be a preferred way to code the lexer. Just don't sprinkle a bunch of unconnected `new RegExp()` / `re.exec` around the code and expect that to be reach peak lexing speed.