The representation GPT-3 learns are flexible. It can learn very complex tasks, including logic and simple arithmetic. The issue is GPT-3 learning algorithm and it's learning capability.

Typically the ability to learn << ability to represent in most cases. Universal approximation theorem type results don't say anything about learnability. GPT-3's abilities quickly fade once it gets outside the distribution it was trained with.