I agree, this really isn't surprising. They're pattern interpolators, and so much of what we deal with in the world is rooted in patterns of some type, including neural nets.

IMO, this is a similar behavior that we saw from AlphaFold, which also used a type of Transformer architecture to make its protein predictions.