To understand the argument you have to allow that the Gary Marcus neural network has a fixed bias towards LLMs being rubbish and so tends to confabulate these sort of things. (Geoffrey Hinton is quite funny on that https://youtu.be/d7ltNiRrDHQ)