I would be more impressed by that paper if they didn't make trivially falsifiable claims: https://twitter.com/gwern/status/1280204127876808705

I am happy to take them at their word that their theory about symbol grounding proves that no LM will ever be able to solve "Three plus five equals" (appendix B); and thus, by modus tollens, GPT-3's ability to (already) solve "Three plus five equals" means their theory is wrong and I need not consider it any further.