I see them collapse when faced unusual problems. They’ll go down a weird rabbit hole and then just keep going, even if their idea is completely irrational and not working.

They’re incapable of changing their mind, because they’re a Markov chain. They’re cool and they’re good at doing things, but they’re not sentient in the slightest.

Not at all the same idea. People also rathole into doomed approaches. What we're talking about here is literally LLMs producing line noise, or English words that don't fit into sentences.

(I don't think LLMs are "sentient".)

You haven't accidentally sent a prompt to a locally-running LLM using the wrong delimiters/formatting then. Noise, random parts of words, looping repetition, it all happens.

Sure, I have no idea what you're running locally, just assume I'm talking about frontier models. It doesn't matter to the argument, because the "stochastic parrot" claim --- here and in Gebru's paper --- is categorical.

I've seen this reflex quite a few times now. To diminish the impressive capabilities of the latest LLMs, one resorts to the assertion that they're not sentient, or not truely intelligent, or various other semantic diversions. No one claimed that. GP was all about usefulness.

Then GP misrepresented the "On the Dangers of Stochastic Parrots" paper, which spoke about meaning. But GP also said "meaning" several times, so no.

There's some haziness here about the difference between comprehensible output and comprehending. To put it mildly. I mean that's been the whole question for five or six years.

LLMs collapsing into piles of gibberish is a misrepresentation, too. The paper doesn't mention it and the argument doesn't hinge on it, or even benefit from it, really. But the other bit:

> Her argument claimed that while we perceived legible text coming from ChatGPT, we were actually being fooled by our innate pattern-matching instincts

This is an accurate summary, I think, and the point the authors are making sits somewhere in the coordinate space of a simplified reframing of the Chinese Room, Grice's account of intention in meaning, Wittgenstein's "Say 'It's cold here' and mean 'It's warm here'" challenge, and a whole bunch of the hermeneutic tradition.

Very roughly, you might say that an LLM doesn't mean anything when it emits text, but that the text still has meaning (which I think is all tptacek wants from "meaningful outputs", but you could take that further as culturally or scientifically 'meaningful', i.e. 'significant').

Philosophically, there are people who hold the position that something has to have been meant to have meaning, or that something has to have meaning in the first place to be meant, and so on. There were always interesting edge cases to these accounts (actors delivering lines, or the Dada/Burroughs cut-up technique), but LLMs are challenging in a way that e.g. Racter wasn't.

"Stochastic parrot" is a fitting term for this because the parrot's words have meaning, but the parrot itself doesn't mean anything by them. Not literally, anyway. And that's it. That's the whole analogy.

Have some people watered the term down to "LLM dumb"? Yes. Is that what Gebru meant in the first place? Not in my reading. Is it incorrect to use the term today? Depends, uh, on what you mean by it.

To be clear, I did not claim that the paper said that; just that it was in the zeitgeist when the paper was released.

I shouldn't have borrowed 'misrepresent' from the person I replied to; 'misunderstand' or 'misinterpret' would've been a better choice. My read of the paper is that you can posit a stable model with perfect recall capable of doing genuinely useful things and it doesn't meaningfully affect the argument, which is more philosophical than technical.

I don't mean to defend the paper in general, though. It has plenty of problems.

I really wish the article in question hadn’t used that phrase, because you’re right. But the rest of the article is also correct in the dangers facing us from this technology as an accelerant regardless, and I think it’s much more interesting to talk about that: but this entire comment thread will dunk on the stochastic parrot paragraphs instead.

The funny thing is though, I think it’s an in group signalling mechanism. You need the anti-AI crowd to know that you’re anti-AI, and this is a rapid way of doing so, to get it upvoted here.

The article is extremist nonsense where the author argues billions are going to die due to climate change in the next decades, and that Altman and Musk are hoping AI can until then replace human labor in order to keep their living standard in such a future.

It's not like the "stochastic parrot" thing was the only problem here.

Sure, if you think so, but discussing or arguing for/against those points here is far more interesting than the done-to-death stochastic parrot stuff.

I’m glad the article used the phrase because it was a useful indicator of the author’s underlying biases

I couldn't read past the first paragraph

> I mean, technically speaking, a video game from 1980 was likely to contain more logical branching points than a modern LLM's kernel, and was likely to be a lot more interesting to read.

Are you really saying that the "AI" in 1980 is _Really_ much more interesting than today's LLM? The author is either dishonest or ignorant.

An LLM did not do those things! An agentic harness around an LLM did those things.

This is a distinction that only matters if you're having a philosophical argument, but the claim I'm addressing from this article isn't philosophical.

It is a practical distinction. An engine is not a car.

For a mechanic, that's a practical distinction. For most people, it isn't.

I don’t see people driving engines around, since that is impossible. You actually need the rest of the car around the engine to have it function as a drivable vehicle.

It is a pretty practical distinction.

Of course. But most people don't buy an engine, ever. They don't have one laying around. They buy a whole car, every time.

So, since apparently I may need to clarify my actual point: For most people, an LLM-plus-agent-harness is just "LLM" or "AI". If you're trying to build one, you care about the distinction. Ditto if you're a pedant. But most people are not either of those things.

We aren’t doing the general public any favors by blurring these distinctions. People can mostly understand that a car usually has an engine, but in this context people can’t tell how the box is moving.

the most widely used API surfaces are already shaped around chat, it's different and most analogies will fall short in some way

Neither a chat or an API are an LLM.

LLM's have chat templates that sit at the lowest levels of how we feed tokens to them. It's quite integral to how they are trained to process tokens and rises up through the tech stack.

Go to a huggingface repo, look at the files, they have a literal chat_template.jinja

Are you saying an engine with a steering rack is a car?

By that standard Gebru was also not writing about LLMs in the Stochastic Parrot paper.

Well, that may be what it was coined to mean, what does it mean now? Stochastic = statistical, parrot = using training data.

https://en.wikipedia.org/wiki/Stochastic_parrot

"a metaphor that frames large language models as systems that statistically mimic text without real understanding". I suppose you're defending them as having understanding. It's not "safe to say that argument has collapsed utterly", you're part of a campaign to reject the meme. On the other side, the meme is popular as a way to be rude about AI, because it's a cutting insult.

> what does it mean now?

It means whatever you want it to mean, that's the best part. To me, it's best used as it was intended, but primarily in the contexts where the stochastic parrots are plowing through the ivory towers we built mostly for idiot savants.

I don't even think the claim (about what she was saying) is true. It's still used with the original meaning, which was not about being fooled into perceiving legible text, but fooled into perceiving understanding.

Sherry Turkle is far more articulate on these things in her latest book Artificial Intimacy (which is about much more than the relationship chatbots)

> It's safe to say that argument has collapsed utterly. Modern LLMs virtually never collapse into endless babbling loops. Far more importantly, they solve objectively hard problems.

And yet, they still output gibberish. They also still fail to solve objectively easy problems on a regular basis. That is because they are, indeed, still stochastic parrots without any actual understanding of anything or ability to reason about stuff. The argument hasn't collapsed at all.

> They also still fail to solve objectively easy problems on a regular basis

Which?

What is exactly "actual understanding" . If it tells 1+1 = 2 it can perfectly explain why the result is 2. Or do you mean the actual transformer work during the 1+1 question?

How'd your parrot do at IMO this year?

LLMs are certainly stochastic, but they no longer meet any sane person's definition of "parrot." I can't imagine Gebru disagreeing. Has anyone bothered to ask her?

(Consider responding with an actual counterpoint, so we can all see what reasoning might give rise to such a delusion. Does your pet parrot do calculus and write stories and reverse-engineer Adobe applications?)

LLMs get better and better and do more and more things. But I still get the impression "their thinking isn't human thinking" by the way that their ability "understand" something shifts and fades as the discussion continues. And these can be things that it previously explained to me.

When I look at a person moving in front of me, I don't have to recheck if their bone structure is going shift as they more - it just won't. So, as a human, I don't naturally notice things of that sort. So I don't notice all the ways an AI video is wrong unless I concentrate. Which is saying that LLMs are wrong in ways we aren't accustomed to. They drift in their interactions and seeming understanding. Perhaps "random walk parrot" is better than "stochastic parrot" but I still say a word that communicates that they're "things that aren't quite doing what you think they're doing" is legitimate.

> is saying more about you than about the models

It's interesting that you'd be choosing this canned phrase of all canned phrases, given just how hard that is linked to emotion-driven facebook comment fights that is.

This doesn't read like something that belongs into an engineering-driven discussion.

I don't understand what you're saying. I feel like I'm making an unusually engineering/science-based argument here: that the term "stochastic parrot" was introduced to describe a constellation of phenomenon that have been more or less decisively disproven, and that people continue to name-drop it as if it were a substantive argument when it's really pure vibes at this point.

To add on, importantly, when Gebru called them stochastic parrots, CoT technique was not published yet, and LLMs had finite computational depth per emitted token, and that architectural limitation was a large contributor to hallucinations. But if you give a sufficiently general LLM file read/write (or arbitrary memory storage) and CoT before having to output a token, it is Turing-complete. Architecturally in principle you can have models that are accurate on studied domains, the question being how to grow a model that achieves this by reading literature and cheap RL on experiments, rather than curated design.