Saying that an LLM can or can't do something because it's a "token predictor" is a category error. The interface isn't a hard limit.
And that's before all the post-training comes in.
What's the "limit" there?
I'm not sure how you're defining "intelligent", but I'd like to know how it is able to exclude a language model, while still including humans, without simply defining it with an axiom that predefines LLMs as lacking intelligence.
An LLM does not persist in consciousness from one token to the next. Each generation, happening hundreds of times a second, will be initialized, generate an output, and terminate. Humans are not stateless like an LLM.
> An LLM has a fixed number of ways it can express itself
While deterministic, there is not a fixed number of ways it can express itself, given that we can use settings like temperature to inject randomness into the output.
> An LLM does not persist in consciousness from one token to the next
While a model alone does not update itself to persist some form of history, there are a number of ways to overcome this, e.g. episodic memory, fine-tuning, and other self-improvement systems exist, which can indeed carry forward what you've called "consciousness".
> Humans are not stateless like an LLM.
A single LLM might be stateless, but an agentic system that relies on LLMs is very often not.
You're missing the point, which is that no matter the process involved. The LLM can only ever output one of the tokens in its token vector. It can't invent a new symbol or character. It can't leave and go build a church. It has to output a little piece of data for you.
Either way, if you consider animals, LLMs are even more poorly positioned. They can do exactly none of the things my cat can do. An LLM can string together words, but if my cat is intelligent, it's clear that stringing together words is not synonymous with intelligence, since my cat can't do that.
It was also not considered to be a different route to the same thing, but more like fraud.
Also conceptually I could just write the weights on paper and do the billion multiplications on paper without any computer, does that mean I am the paper or the numbers or what??
Like I said, "without simply defining it with an axiom that predefines LLMs as lacking intelligence"
Contrary to popular belief, training a LLM is not just about memorization (overfitting). There is some memorization happening, but well-trained LLMs also generalize.
Doubting the sentience of machines and even other humans is perfectly fine though. Only empathy allows people to make the leap and assume other humans have souls.
Any leap beyond that is based on empathy. You have a soul, and you are human, therefore other humans could have souls too. It's a spiritual belief. Answers to questions that cannot be answered.
Developing theory of mind is one of the key milestones in child development.
> It's impossible for someone to doubt their own sentience. The literal act of doubting is enough to dissipate all doubt.
i never found this convincing. just because you can loop does not mean you are sentient/conscious. what would it look like if you didn't exist and there was just a system that interrogated neural inputs and produced neural outputs in a loop? if anything, LLM's as an existence proof made this more likely to be the actual case. consciousness could literally be nothing more than just a sidekick that enriches the existing context with analysis, then when "you" take an action, just like an LLM it's forced in a new direction.Not I. I'm just a Boltzmann brain.
> LLMs appear to learn distributions of representations, they both develop a hierarchy of those representations, both have early layers that process simple features, with later ones processing more abstract concepts, both predict missing information.
This type of superficial comparison isn't very meaningful, it's trivial to liken anything to a human biology in this manner.
A plane and a bird both use wings to produce lift, it doesn't then follow that a bird and a plane are meaningfully similar.
The use of Bernoulli's principle to achieve lift is a fundamental and meaningfully similar function of both airplane and bird wings. That functional similarity is well known.
> This type of superficial comparison isn't very meaningful
The comparisons I provided are fundamental to both the human mind and LLMs.. that's pretty darn relevant.. and whether you find that trivial or not is a matter of opinion.
Natural language is, however, a prerequisite for the existence of LLMs. It’s more similar to methods for storing and retrieving information, like the printing press or a database, than it is to a sentient being.
That’s not to say that LLMs can’t do crazy things, because they already have. Our language can encode a whole lot of information, and it’s incredible that we’ve found a way to distill that so effectively.
I think they’re not talking about cognition, but about output: regardless of what may be happening inside your brain, ultimately one word at a time comes out of your mouth, right? And you can’t then unsay it.
When you put it in those terms, LLMs are in exactly the same boat.
You're conflating being autoregressive with being sequential.
But right now I am trying to design something -— a physical mechanism with a particular enclosure — that I cannot clearly describe (this makes it hard to research). I designed a previous version without even knowing the words that do, in fact, describe that.
I have a theory about it, animated in my mind, that I can only test by making it.
If I want you to know about it, I can either show you it or work out words to describe it, which will be inadequate to describing it.
The idea for it came from seeing things nobody has ever put into words for me.
"Next-word sayer" doesn't describe any of this process, does it?
(This is also why text-to-CAD is a bullshit idea)
And that has a limit. If you are stuck at PoC level or simple apps, you have no idea how limited the current models still are. There you really need to break tasks down, not just trust a token predictor to list steps that sound good. There has to be a human in the loop somewhere, because by the time you start skipping permissions, best case you get the jackpot, more likely is you get a suboptimal solution and token waste and what's genuinely still terrifying when the model ignores instructions and does some stupid nonsense, ruining your day. It really is as sharp as a CNC machine. It's not not useful, but could be dangerous, so maybe don't try to carve wood with a monster machine, or park your Ferrari in that crammed neighbourhood if you don't know how to parallel park.