By the way, would you say that Stockfish 15 understands how to play chess well?
Of course it’s a different question as to how exactly do humans understand chess.
I personally don’t believe humans have some sort of “special” intelligence, whatever we do must be computable and therefore doable by a computer. It’s just that todays LLMs don’t seem to be it.
related: https://parrotchess.com/
Meaning that for computer programs they're very bad at chess.
I've been thinking about this and I think there is a simple trick to make LLM's play chess well. Any chess position can be represented by a string, like so: rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1 (Forsyth–Edwards Notation) If we think of a chess game as a sentence or a sequence of such strings or tokens, then after feeding tons of those to the model it is reasonable to suspect it will do what it does best, that is predict the next token. There is no spacial awareness or anything beyond that going on I think.
https://twitter.com/GrantSlatton/status/1703913578036904431
I have tested it myself.
1- unlike language where you can make an argument that maybe model had seen this example in the training set and just parroting it back, you can't really memorize the possible states of chess and thus you can be reasonably sure that any game or state it is in, must be OOD.
2- If the system still plays well with OOD examples, it must be gaining understanding of the game itself.
Also check out the Othello paper, where they show the evidence for an LLM encoding the game state in its neurons.
See this tweet.
https://twitter.com/GrantSlatton/status/1703913578036904431
I have tested it myself.
Obviously you can define words like 'learn' to exclude AI training but there is no associated benefit to clarity.
While LLMs are “merely” token predictors, it seems that perhaps the “knowing” is actually imbedded in the large corpus of memetic data.
In the same way that an infinitely detailed “choose your own adventure” book shows that data and computation are interchangeable by degree, (calculation vs lookup), my conjecture is that the often surprising effectiveness of LLMs, as well as there very human like flaws, are the result of this kind of computation-in-the-data scenario.
I posit that the overt connections in textual human knowledge also carry the implied knowledge that provides the unwritten context necessary to understand, if enough vector relationships can be teased out of a sufficiently large corpus of data.
It could easily be that our own “understanding” is also derived from the vast n-dimensional matrix that represents all human cultural knowledge.
For a though experiment, you could put a human in a room, and only communicate with them via a terminal. You can write them messages, but the only way they can communicate to you is by giving you a probability for the next token of their answer, then you pick one with a method of your choosing, tell them what you picked, and they choose the next probability. This would severely hamper their ability to communicate effectively, and somebody who only sees the interface and doesn't know the probabilities might conclude it's "only a token predictor". But is it any less of a general intelligence because of that? After all it's still a human, just with a "dumb" input/output protocol that happens to be a token predictor.
The method signature at the end doesn't matter you are right there but what the model is trained to do matters a ton. If it was trained to solve logical problems or to fact check etc, then it is no longer just a token predictor. But as is all the model is trained to do is to predict tokens, and you notice that immediately when you use it because it leads to obvious issues with what the model can do. It is really easy to target sparse parts of its data and get it to say nonsense because it is just trained to predict tokens.
You're being loose with semantics and drawing bad conclusions. The training describes the nature of the feedback given to the system. Being trained to predict tokens means it gets feedback on the quality of its predictions. This does not constrain or determine the structure of the learned model. The learned model is a function of the training time, quality and amount of data, as well as the architecture that is biased towards capturing certain features or entails limits on what can be captured. The point is, the space and properties of solutions to the problem of "predicting the next token" is not constrained by the nature of the feedback.
It absolutely does! A human can learn to perform an action by reading a manual. But an LLM is only trained to predict tokens, if you feed it a manual it will learn to recite the manual, it wont learn how to perform the actions in the manual.
This makes the training inherently flawed from a learning perspective, and that flaw is impossible to fix without moving past the token prediction way of training models.
This claim doesn't track with LLM-based agents. For example: https://towardsai.net/p/machine-learning/meet-webagent-deepm...
Unfortunately, I’ve met several humans to which this statement would also apply.
So like a marketing department? Or most humans, most of the time, for that matter.
No... It detects temperature and reacts. It does not feel pain. Feels implies both affect and sensation, both of which are experiential, phenomenological qualia requiring some corollary of a nervous system and sentience, neither of which exist within the underlying structure of an LLM or indeed any current AI. AI can no more feel pain than a sliding door can decide to open. It's deterministically reacting to stimuli.
And this is when studying actual living organisms, so applying it to machines will be even more useful.
Go read this discussion I had with GPT-4:
https://gitlab.com/-/snippets/2594242
I wanted to get a feeling for the distinction philosophers used between "syntax" and "semantics", as well as exploring what someone in that camp would say about "concepts" and "understanding". The conversation seemed rational; just to make sure it wasn't way off base, I sent it to a friend of mine who had trained in philosophy, and he said he agreed with GPT-4 (and specifically, he didn't say "that's not what those terms mean in philosophy at all").
Now consider the sentence:
$SUBJECT $VERB the philosophical $NOUN of syntax and semantics well enough to apply it to my example of an adder circuit.
If $SUBJECT was "my nephew", then the most natural words to use for $VERB and $NOUN are "understood" and "concepts". What words should we use when $SUBJECT is "GPT-4"?
I submit to you that the most natural words to use in the context remain the same.
And in fact, "Of all places on the internet", I'd expect a forum filled with programmers to be very "functionalist". We spend our entire lives bridging the gap between "syntax" and "semantics" -- taking mechanical interactions and creating meaning out of them.
> Of all the places on the Internet, this is where I expected to see most people with minimal AI knowledge.
It should be. However, you're not far from a ChatGPT wrapper project that needs to get enough hype and attention to get the VC funding they need to execute their side grift.
Any 'technologist' that knows beyond the basics of AI knowledge will find that they don't trust these LLMs since they can't reason or understand their outputs and just spit out nonsense. This paper shows it is not even comparable to humans. It is not the same thing either.
Only a matter of time to search for an adversarial prompt to confuse these systems reliably.
LLMs just predict tokens. It's truly amazing how much value you can get out of that, but we don't need to make more out of it than is necessary.
These models were trained on an unfathomable amount of text generated by real humans, we shouldn't be surprised that they sound convincing. But we also shouldn't confuse their ability to sound convincing with legitimate intelligence.
If anything, it seems like we're learning that the Turing test isn't a good predictor of actual intelligence. Mostly because real humans are really easily tricked into seeing intelligence where there is none (and paradoxically not seeing intelligence where there is).
Out of curiosity, how do you think human intelligence works? I am no expert and I don't claim to know how the human brain works but my layman's mental model of thinking would be basically that: biological neural networks that "do" the thinking are token prediction machines, except "tokens" are not words that appear on the screen but thoughts that appear in consciousness... while the underlying machinery is in both cases a network of units that fire (or not) based on the connections between them.
Sure, human experience is a lot more than just intelligence (I have no mental model of how consciousness or qualia might work) but it is surprising to me that so many people keep repeating arguments similar to yours - that LLMs do not really think, they are "just" doing [X] (where X usually describes how I imagine human intelligence works) - if there is more to human intelligence, what do you think it is?
if there is more to human intelligence, what do you think it is
A gut feeling that one thing is kind of analogous to another is just that.
A carpenter and a woodpecker both bang on wood, but knowing anything at all about one doesn't really tell you anything about the other.
I suspect laypeople would be less likely to try to make the connection between LLM graphs and physical neurons, if they wouldn't have called those groups of equations "neural networks".
They share the name, but not much else.
An LLM playing chess is only able to do so because it essentially encoded what to do for any given board state by having seen a lot of chess games. That's why it's fairly competent at the beginning of a game when the number of possible states is low, but falls apart in later stages where memorization is useless. That's why LLMs make illegal chess moves, generate code that doesn't parse and hallucinate. They have no mental models, no understanding.
A human playing chess would instead slowly build up a mental model of the game as they played and play off that model.
... maybe, I dunno.
Imagine that I am building a translation app: one way to do it would be simply a dictionary database where the Czech word "pták" is related to the English word "bird"- we certainly agree that this app does not understand what a bird is at all.
But when I say I do understand what a bird is, what does that mean? The way I imagine it, my brain is wired in such a way that when the word "bird" occurs, a certain pattern of neurons that encode stuff like "living creature", "flying", "feathers", "eggs", and "beak" also fire (each of those is itself a pattern of other associations).
In this sense, I think LLMs can "understand": connections between words "bird" and "eggs" or "flying" have weights closer to 1 while connections to words like "iron" or "multiplication" have weights closer to 0. A mental model of a bird can be represented as a set of vectors in multi-dimensional space.
Maybe I am missing something but I see no fundamental reason why we would not be able to encode complex mental models like chess or liberalism or flat earth theory as sets of vectors (and maybe even determine the set of operations that can move someone from flat earth model to globe model in a similar manner as those examples with 'King – Man + Woman = Queen').
Again, I am not saying I know how the human brain works. All I am saying is that I think we can plausibly explain a lot of it in terms of just networks of units that have some weights attached to them - it is a very powerful concept. That is why I am asking when people seem to be assuming that there is more to "legitimate intelligence".
Also - and I say this with a high level of insecurity because I am no expert - I do not think that LLMs actually play chess the way you describe it. If I even understand correctly what you mean - are you saying that LLMs are only able to play moves they have "seen" in their training sets? If that is the case, I would disagree - I think there is evidence that LLMs have some ability to generalize, which means the ability to work with abstract patterns (which I would be tempted to call mental models).
I'm in favor of duck-typing these words. If I would have called it learning on a human, I can call it learning on a complex neural networks who's inner workings we don't fully understand.
Hard disagree. A subset of behaviours resembling or even "indistinguishable" from understanding should not be enough to meet the definition.
Maybe some day we will have AI systems that get there, but LLMs ain't it.
Nope, it doesn't work this way. One doesn't get to sit and try to say what something isn't w.r.t a 1st party experience (like understanding or consciousness), without saying how a 3rd party can define what it is.
There is no valid argument for "this isn't understanding because my gut says so" without giving a way a person (a 3rd party to an llm) can say what understanding is. We will never have a 1st person "perspective" of a machine LLM.
So if you want to claim as a 3rd party what understanding isn't, you need to define, using tool available to a 3rd party observer, what it is beyond "when my gut says so".
And if you're defining something by external observation, you are duck typing.
Put succinctly, the only practical (non- philosophical) definitions of consciousness, understanding, and intelligence will all be done via duck typing. As no one will ever have a 1st person view of these phenomena for other entities.
But you wouldn't call that learning as a human. If you include a chess manual in the learning corpus of an LLM, but not any examples of playing chess, then the LLM can give you all the rules about chess, but it can't play chess. That isn't learning, that is just parroting.
A human who can recite the rules of chess but can't play chess, would you say that he understands the rules of chess? No, he just know how to output the words and not perform the actions, so there is no understanding there. Same with the LLM, nobody can say it understands these things since you would never said a human understood under similar conditions.
If you sat someone down who had never played chess before, and taught them the rules, they also can't play chess. They wouldn't remember the allowed moves and would make tons of illegal moves.
The only way a person reaches the state of being able to play chess is to usually have learned the rules and also watched a bunch games. Either games between others, or most frequently games of "almost chess" that they play with a helpful instructor that corrects them every time they make an illegal move.
In that respect both humans and llms learn to play chess the same way - by watching games.
Or for an analogy, think about how many times you've sat down to learn a new board game and after the person who had played it before explains all the rules you still don't really get it. What's next? "Let's just play a practice round and I'll pick it up as we go." Ie learning from seeing the game played.
I think a lot of the skepticism in LLMs is people over estimate / inaccurately remember how people really learn / behave.
They can recite the rules perfectly, hence they remember them, I didn't say he just saw the rules once, both the LLM and the human has seen the rules enough to recite them perfectly.
Then a typical human can start to play chess based on those rules without having seen anyone play before, humans learn to play games by reading manuals all the time. You trying to argue against it here is absurd.
> think about how many times you've sat down to learn a new board game and after the person who had played it before explains all the rules you still don't really get it
I read the manual before playing board games because I like reading manuals, it is me teaching my friends then. You don't need someone to show you how to play games, reading the manual is enough...
Maybe you need two models to understand things, since the models wont understand itself but a model can understand the data encoded in the next model. This isn't an unfixable problem, but it is a problem that has to be solved.
[1]: https://www.reddit.com/r/naturalism/comments/1236vzf/on_larg...
For example, if you train an LLM with a chess manual but no chess games then the LLM can recite all the rules of chess, but it wont be able to play chess. That is what people mean when they say "LLM's are just token predictors, they don't understand anything".
So it has nothing to do with spirituality or consciousness or metal vs brains, its just a logical argument based on the limitations of just training the model to predict tokens to match data it has seen. Since it only tried to predict tokens it has seen, and it has never seen a chess game, it doesn't matter if it has seen all the rules, it was never trained to try to follow rules it sees so it can't play chess.
This is just to repeat the initial assertion with more words. What arguments do you have in favor of this claim? The example you give doesn't track with LLM-based agents. And I'm sure I've come across demonstrations of GPT-4 playing a made up game given the rules of the game, but I can't put my hands on it at the moment.