I read through that and none of the section (or entire work) ever talk about the above discussion. Further I looked at some of the many citations of on that section and none of them suggest that the OP is right. In fact a few of them I know disagree.

Depends. The current generation of models rely on a matrices of floats and so there's little chance an umbrella-term like "stochastic parrot" will never not stick. If you are like me (and them: https://archive.is/cZi83) and deem instruction following, chain-of-thought prompting, computational properties of LLMs (as researchers continue to understand training and scaling better) as emergent, then we're on the same page.

Okay so just to confirm that section doesn't actually tell us anything about this and in fact this is all based on your own understanding of the mechanisms involved.

My reading of the papers is, given enough scale, modality, and memory; there are chances (perhaps newer and different) models will be able to "generalize" our world. Also: https://archive.is/3yyZZ / https://twitter.com/QuanquanGu/status/1721394508146057597 | And: https://archive.is/bW2tS / https://twitter.com/mansiege/status/1680985267262619648