I'll admit I haven't seen the training data but some basic googling shows a subset of the labeling is syntax annotations. I am not claiming LLMs parse code in the way you are suggesting, but they certainly have token level awareness of syntax and probable relations which are the roots of any programming language.

Last time I researched AST parsing was quite rudimentary for LMs. The problem was preserving their structural properties, whereby a flattening approach via node traversal tended to remove. But you needed to do to put it into a format that language models could parse.