The purpose of tokenizing everything is to report all errors upfront. Imagine a big source file with two syntax errors. Tokenizing and parsing everything and reporting both errors with one compilation is a better user experience.