> Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory to generate the AST

And some don't even generate an AST. :) Just read in and emit or interpret.

https://briancallahan.net/blog/20210814.html