You got many answers, but I'll give another brutally honest answer: I do this because it's simply much much easier to implement. I would much rather have my AST generator's input be a token stream. It just simplifies everything.