> Her argument claimed that while we perceived legible text coming from ChatGPT, we were actually being fooled by our innate pattern-matching instincts
This is an accurate summary, I think, and the point the authors are making sits somewhere in the coordinate space of a simplified reframing of the Chinese Room, Grice's account of intention in meaning, Wittgenstein's "Say 'It's cold here' and mean 'It's warm here'" challenge, and a whole bunch of the hermeneutic tradition.
Very roughly, you might say that an LLM doesn't mean anything when it emits text, but that the text still has meaning (which I think is all tptacek wants from "meaningful outputs", but you could take that further as culturally or scientifically 'meaningful', i.e. 'significant').
Philosophically, there are people who hold the position that something has to have been meant to have meaning, or that something has to have meaning in the first place to be meant, and so on. There were always interesting edge cases to these accounts (actors delivering lines, or the Dada/Burroughs cut-up technique), but LLMs are challenging in a way that e.g. Racter wasn't.
"Stochastic parrot" is a fitting term for this because the parrot's words have meaning, but the parrot itself doesn't mean anything by them. Not literally, anyway. And that's it. That's the whole analogy.
Have some people watered the term down to "LLM dumb"? Yes. Is that what Gebru meant in the first place? Not in my reading. Is it incorrect to use the term today? Depends, uh, on what you mean by it.
I don't mean to defend the paper in general, though. It has plenty of problems.
There's some haziness here about the difference between comprehensible output and comprehending. To put it mildly. I mean that's been the whole question for five or six years.