To elaborate a bit on that, exposing an AST directly means that you're tied to that version of the AST; this was considered really hard for compatibility over versions. Token streams are much easier to define an interface for, and more flexible: you're not just stuck with whatever AST representation we'd have given you.

For example: https://doc.rust-lang.org/book/first-edition/procedural-macr...