Language models are not grounded learners. The language produced does not really correspond meaningfully to our world except in superficial (albeit complex) ways.
Do you have thoughts on how to move forward on this problem? Maybe ask GPT-3 and see what it thinks :P
And while this is possible, it feels there should be more effective ways to impart a knowledge of reality- if only we had huge databases of usable data to feed to these NNs instead of dumps of text. At the moment it feels like we're trying to teach advanced physics to a subject with no previous knowledge of physics or math by just feeding it with everything on arXiv and physics textbooks in random order. What you get is someone who can produce text that mimics the superficial style of scientific articles, but with an extremely confused understanding of the subject, if any at all.
I am happy to take them at their word that their theory about symbol grounding proves that no LM will ever be able to solve "Three plus five equals" (appendix B); and thus, by modus tollens, GPT-3's ability to (already) solve "Three plus five equals" means their theory is wrong and I need not consider it any further.
While it's true that recent natural neural net models like ixvvqktiwl may sound superficially coherent and like they 'understand' things, we can see by comparison with artificial neural net models that they aren't really doing anything we'd call "natural intelligence"; it's advanced mimicry at best, just elaborate pattern matching.
I get that it's very easy to create these natural neural net models and be carried away by excitement, and it can even be profitable (witness the many VC-funded startups which use natural neural nets as a core technology), but we should remain skeptical of any claims by those natural neural net models, much less their promoters online, that they are 'intelligent' in the strict definition of the word.