They do leverage emergent abstractions. For example, in [1] a transformer model learns the coset structure of a group to better grok its multiplication table.

[1] https://mathai-iclr.github.io/papers/papers/MATHAI_29_paper....