We wanted a parse tree that could be serialized out to produce the original document byte for byte, but still reflected the grammar.
Things get uglier for JSON in the real world case:
"5+7 #comment"
becomes: <sum><integer>5</integer><operation>+</operation><integer>7</integer> #comment</sum>
The XML output is essentially the original input + grammar markup. Your solution also works but is a data structure representation with a lot more line noise for humans to read.This was primarily a mechanism to load parsed documents in to code that didn't have access to the recursive descent parser that parsed the document, for either security or performance reasons.
Loading the XML with the appropriate parse flags, doing a pre-order traversal on all the leaf PCDATA, and then printing it out, would regenerate the original document byte for byte.
Perhaps I should go back and implement JSON output and see how I feel about it... thought I'm pretty sure once you introduce a bunch of "text" and "type" fields to the JSON tree it's going to look a lot less readable than XML markup.
There are other fields as well like line and column numbers (output as XML attributes)
I've had to roll my own hack for such cases.