> Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory to generate the AST
And some don't even generate an AST. :) Just read in and emit or interpret.
> Some compilers tokenize while parsing, but for a different reason: it's faster and uses less memory
> Most compilers try not to fail when they encounter a syntax error, they try to "recover" and parse the remaining document
"> It seems akin to natural language processing. You'd interrupt a speaker early if you can't make sense of his uttering."
But if you're reading a book and can't understand a sentence, you'll probably glance at the rest of the page to see if there are clues in the context.
Maybe it's a printing error and a paragraph has been repeated. A human who sees the entire page will detect that problem immediately and simply skip over the repeated paragraph. A computer that gives up at the first sight of incongruency has no idea that recovery was so easy.
The analogy here might be relevant to error messages. A compiler that "sees the whole page" can potentially offer more useful suggestions to the programmer about how to fix a problem.
Most compilers try not to fail when they encounter a syntax error, they try to "recover" and parse the remaining document, usually starting from the next valid statement or declaration. This lets them report more syntax errors and moreover, is very important for good IDE support (if you've ever used an IDE/language combo where you only get completion after you've written the code, you know why). Although if the syntax error messes up tokenization (e.g. missing quote or end comment) it usually screws up the rest of the document anyways.