I certainly agree that GPT-3 appears to be learning how to do mathematics. I suspect that if you gave it enough it might perhaps even learn the maths of physics.

I suspect that if it did that, it would be able to write a very convincing fake paper about how it designed and tested an Alcubierre drive, and that the main clue about the paper being fake being a sentence such as “we dismantled Jupiter for use as a radiation shield against the issue raised by McMonigal et al, 2012”.

Or, to put it another way, the hardest of hard SciFi, but still SciFi, not science.