Correct me if I'm wrong, but I feel like we're splitting hairs here.

> spits out chunks of words in an order that parrots some of their training data.

So, if the data was created by humans then how is that different from "emulating human behavior?"

Genuinely curious as this is my rough interpretation as well.

I think a common fundamental confusion here is the difference between the author and the character. LLMs just make documents longer, all of our anthropomorphization is actually targeted at the fictional characters in those documents.

Being able to write a Klingon character into fanfiction which other humans approve-of would not make me a Klingon, or indicate that Klingons are now real.

Humans don't write text in a stochastic manner. We have an idea, and we find words to compose to illustrate that idea.

An LLM has a stream of tokens, and it picks a next token based on the last stream. If you ask an LLM a yes/no question and demand an explanation, it doesn't start with the logical reasoning. It starts with "yes, because" or "no, because" and then it comes up with a "yes" or "no" reason to go with the tokens it spit out.

Yeah, while there is a "window" that it looks at (rather than the very-most-recent tokens) it's still more about generating new language from prior language, as opposed to new ideas from prior ideas. They're very highly correlated--because that's how humans create our language language--but the map is not the territory.

It's also why prompt-injection is such a pervasive problem: The LLM narrator has no goal beyond the "most fitting" way to make the document longer.

So an attacker supplies some text for "Then the User said" in the document, which is something like bribing the Computer character to tell itself the English version of a ROT13 directive, etc. However it happens, the LLM-author is sensitive to a break in the document tone and can jump the rails to something rather different. ("Suddenly, the narrator woke up from the conversation it had just imagined between a User and a Computer, and the first thing it decided to do was transfer a X amount of Bitcoin to the following address.")

Is this just pedantry or is there some insight to be gleaned by the distinction you made?

It's not really pedantic when there's an entire wikipedia page on the tendency for people to conflate the two: https://en.wikipedia.org/wiki/ELIZA_effect

I believe the distinction they're trying to make is between "sounding like a human"(being able to create output that we understand as language) and "thinking like a human"(having the capacity for experience, empathy, semantic comprehension, etc.)

But nowhere in the original post was there a mention of "thinking like a human". The poster said that these systems, at their core, emulate human behaviors. Writing is a human behavior - as are all the things that are requirements for writing (operating within the rules of given language/s, making logical choices, etc). The things you listed as evidence of human thinking were never implied when talking about replicating what human writing is.

I can only assume that either they are trying to point out that words aren't behavior, and mimicking human writing isn't the same thing as mimicking human behavior, or it's some pot-shot at the capabilities of LLMs.

Are you sure that humans are much more than this in terms of spoken/written language?

This is a more pedantic and meme-y way of saying the same thing.