> that gradient descent finds it easier to learn the actual physics, then it'll learn the physics.

I guess it really depends on what the meaning of gradient decent learning the physics is.

Maybe you define it to mean that the actually correct equations appear encoded in the computation of the net. But this would still be tacit knowledge. It would be kind of like a math software being aware of physics at best.