Tokenizing everything in one go is usually the right approach. The typical case is the error-free case, so you would not tokenize more characters than necessary anyway, but you simplify the implementation and avoid jumping back and forth between parsing and tokenizing.