While I imagine alpha go does some brute force and some tree exploration, I think the main "intelligent" component of alpha go is the ability to recognize a "good" game state from a "bad" game state based on that moment in time, not any future plans or possibilities. That pattern recognition is all it has once its planning algorithm has reached the leaves of the trees. Correct me if I'm wrong, but I doubt alpha go has a neural net evaluating an entire tree of moves all at once to discover meta strategies like "the opponent focusing on this area" or "the opponent feeling on the back foot."
You can therefore imagine a pattern recognition algorithm so good that it is able to pick a move by only looking 1 move into the future, based solely on local stone densities and structures. Just play wherever improves the board state the most. It does not even need to "understand" that a game is being played.
> while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead.
So I don't think this statement is necessarily true. "Understanding" is a major achievement, but I don't think it requires planning. A computer can understand that 2+2=4 or where to play in tic-tac-toe without any "planning".
That said, there's probably not much special about the concept of planning either. If it's just simulating a tree of future possibilities and pruning it based on evaluation, then many algorithms have already achieved that.
So trying to make an AI that solves the entire problem before writing the first letter will likely not result in a good solution while also making it compute way too much since it solves the entire problem for every token generated. That is the kind of AI we know how to train so for now that is what we have to live with, but it isn't the kind of AI that would be efficient or smart.
Such answers will be very hard for an LLM to find, instead you mostly get very verbose messages since that is how our current LLM thinks.
It can be instructed to study its previous answer and find ways to improve it, or to make it more concise, etc, and that is working today. That can easily be automated by LLMs talking to each other.
There’s no limit to how sophisticated a model can get, but,
1. That’s a property shared with many architectures, and not really that interesting,
2. There are limits to the specific ways that we train models,
3. We care about the relative improvement that these models deliver, for a given investment of time and money.
From a mathematical perspective, you can just kind of keep multiplying the size of your model, and you can prove that it can represent arbitrary complicated structures (like, internal mental models of the world). That doesn’t mean that your training methods will produce those complicated structures.
With Go, I can see how the model itself can be used to generate new, useful training data. How such a technique could be applied to LLMs is less clear, and its benefits are more dubious.
For an LLM, this is not even close to being the case. The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility. And we certainly haven't figured out a "model" which would let a computer generate new and valid data that expands its understanding of the world beyond its inputs, so self-training is a non-starter for LLMs. What the LLM is "understanding", and what it is reinforced to "understand" is not the world but the format of texts, and while it may get very good at understanding the format of texts, that isn't equivalent to an understanding of the world.
No human or creature we know of has a "true" world model so this is irrelevant. You don't experience the "real world". You experience a tiny slice of it, a few senses that is further slimmed down and even fabricated at parts.
To the bird who can intuitively sense and use electromagnetic waves for motion and guidance, your model of the world is fundamentally incomplete.
There is a projection of the world in text. Moreover training on additional modalities is trivial for a transformer. That's all that matters.
Despite not having the bird's sense for electromagnetic waves, I have an understanding that they are there, because humans saw behavior they couldn't describe and investigated, in a back-and-forth with a world that has some capacity to disprove hypotheses.
Additional modalities are really just reducible to more kinds of text. That still doesn't exhaust the world, and unless a machine has some ability to integrate new data in real time alongside a meaningful commitment and accountability to the world as a world, it won't be able to cope with the real world in a way that would constitute genuine intelligence.
Yeah this isn't really true. There's not how humans work. For a variety of reasons, Plenty stick with their incorrect model despite the world indicating otherwise. In fact, this seems to be normal enough human behaviour. Everyone does it, for something or the other. You are no exception.
And yes LLMs can in fact tell truth from fiction.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
Your argument seems to boil down to "they can't perform experiments" but that isn't true either.
The ~100 layer deep neural networks infer many levels of features over the text, including the concept of true and false. That is trivial for an LLM.
Are you completely unaware these are based on deep neural networks?
Convolutional Neural Networks don't operate by "look up" of text data.
It's response is not based on facts about the world as it exists, but on the text data it has been trained on. As such, it is not able to determine true or false even if the response in the above example would be correct.
I know I don't experience the world as it is, but rather through a whole bunch of different signals I get that give me some hints about what the real world might be. For example, text.
LLM does not, that isn't how it works.
You can say the difference is academic but there is a difference.
What is the difference between a real good faker of intelligence and actual intelligence is an open question.
But I will say most AI experts agree that LLM are not artificial general intelligence. It isn't just a lack of training data, they just are not of the category that we mean by that.
> LLM does not, that isn't how it works.
GPT-4 can explain the concept when prompted and can evaluate logic problems better than most human beings can. I would say it has a deeper understanding of "true vs false" than most humans.
I think what you are trying to say is that LLMs are not conscious. Consciousness has no precise universally agreed formal definition, but we all know that LLMs are not conscious.
Sigh
GPT produces output which obeys the patterns it has been trained on for definitions of true and false. It does not understand anything. It is a token manipulation machine. It does it well enough that it convinces you, a walking ape, that it understands. It does not.
You have a subjective philosophical disagreement about what constitutes understanding. That is fine. I clearly understand it is not conscious and that programs do not understand things the way that humans do. We are fundamentally different to LLMs. That is obvious. But you are not making a technical argument here unless you can define "understand" in technical terms. This is a matter of semantics.
> It is a token manipulation machine
Deep learning and machine learning in general is more than token manipulation. They are designed for pattern recognition.
The broad strokes you use here are exactly why discussing LLMs are hard. Sure some people dismiss them because it isn't general AI but having supporters dismiss any argument with "passes the Turning test" is equally useless.
"But you are not making a technical argument here unless you can define "understand" in technical terms. This is a matter of semantics."
I said the nature of their argument is not technical, since they are not dealing with technical definitions, but I did not dismiss their argument altogether. I clarified and restated their own argument for them in clearer terms. LLMs are not conscious, but they can still "understand" very well depending on your definition of understand. Understanding is not a synonym for consciousness. Language is evolving and you need to be more precise when discussing AI / machine learning.
One definition of understand is:
"perceive the intended meaning of (words, a language, or a speaker)."
Deep learning models recognize patterns. Mechanical perception of patterns. They understand things mechanically, unconsciously.
The core issue is their "knowledge" is too context sensitive.
Certainly humans are very context sensitive in our memories but we all have something akin to a "mental model" we can use to find things without that context.
In contrast LLM has knowledge defined by that context quite literally.
In either case my original point on using true and false is that LLM can hallucinate and on a fundamental design level there is little that can be done to stop it.
Sloppy non-rigorous shallow arguments are bad for discussion.
GPT-3.5 was trained on at least 300 billion tokens. It has 96 layers in its neural network of 175 billion parameters. Each one of those 96 stacked layers has an attention mechanism that recomputes an attention score for every token in the context window, for each new token generated in sequence. GPT-4 is much bigger than that. The scale and complexity of these models is beyond comprehension. We're talking about LLMs, not SLMs.
ChatGPT can tell me about itself when prompted. It tells me that it is an LLM. It can tell me about capabilities and limitations. It can describe the algorithms that generate itself. It has deep self knowledge, but is not conscious.
Your eyes and ears perceive a tiny minuscule fraction of what is out there in the real world.
A blind and deaf person must know even less of the real world than an LLM, which can read more than a human can ever read in their lifetime.
How did you find out that Biden was elected if not through language by reading or listening to news? Do you have extra sensory perception? Psychic powers? Do you magically perceive "facts" without any sensory input or communication? Ridiculous.
By the same argument your knowledge is also not based on "facts" about the world, since you only learned about it by reading or listening. Absurd nonsense.
Do LLMs know true or false? It depends on how you define "know". By some definitions, they "know true or false" better than humans, as they can explain the concept and solve logic problems better than most humans can. However, by any definition that requires consciousness, they do not know because they are not conscious.
The average person spends a lot of time completely immersed in "false" entertainment. Actors are all liars, pretending to be someone they are not, doing things that didn't really happen, and yet many people are convinced it is all "true" for at least a few minutes.
People also believe crazy things like Flat Earth theory or that the Apollo moon landings were faked.
So LLMs have a conceptual understanding of true/false, strong logical problem solving to evaluate truth or falsity of logical statements, and factual understanding of what is true and false, better than many humans do. But they are not conscious therefore they are not conscious of what is true or false.
And I have no idea what you mean by saying that it has no concept of true or false. Even the simplest computer programs have a concept of true or false, that's kind of the simplest data type, a boolean. Large language models have a much more sophisticated concept of true and false that has a lot more nuance. That's really a pretty ridiculous thing to say.
Tell me how that doesn't apply to the human brain as well.
If you don't have a world model then you are a vegetable and could not be replying on HN.
If you're taking out the recycling, do you take the time to identify (model) each piece of it first? No, because that's not necessary.
The vast model in your brain is learned and generated unconsciously without your direct awareness.
But if you're just using it to mean "factual memory", calling it modeling seems like false precision.
If you close your eyes and get lost after a few seconds, that's because your model was not a 100% perfect exact replica of external reality that extended infinitely far in all spatial directions at all resolutions. For example, your model does include the entire surface of Mars, and that doesn't mean that your model does not exist. Models are not perfect by definition. I thought this would be obvious.
The model of reality in your head is a simplification that serves a purpose. Pointlessly closing your fully functioning eyes is not something your model generating hardware was evolutionarily optimized for. Natural selection weeds out that kind of behaviour.
If you become blind then your model generation will learn and optimize for other sensory inputs. Think of a blind man with a cane.
You're using "your model" as a metaphorical term here, but if you came up with any precise definition of the term here, it'd turn out to be wrong; people have tried this since the 50s and never gotten it correct. (For instance, is it actually a singular "a model" or is it different disconnected things you're using a single name for?)
See Phil Agre (1997) on exactly this idea: https://pages.gseis.ucla.edu/faculty/agre/critical.html
David Chapman (more general and current): https://metarationality.com/rationalism
and this guy was saying it in the 70s: https://en.wikipedia.org/wiki/Hubert_Dreyfus#Dreyfus'_critic...
> limited to some degree of accuracy
This isn't the only issue:
- You may not have observed something in the room in the right way for the action you need to do later.
- You might have observed it in a way you don't need later, which is a waste of time and energy.
- It might change while you're not looking.
- You might just forget it. (Since people do this, this must be an adaptive behavior - "natural selection" - but it's not a good thing in a model.)
> Why would you think any model has to be a perfect exact 1:1 representation of the entire universe?
What principle can you use to decide how precise it should be? (You can't do this; there isn't one.)
> The model of reality in your head is a simplification that serves a purpose.
Not only does it serve a purpose, your observations largely don't exist until you have a purpose for them.
RL agents tend to get stuck investigating irrelevant things when they try to maintain models; humans are built to actively avoid this with attention and boredom. Robot cameras take in their entire visual field and try to interpret it; humans both consciously and unconsciously actively investigate the environment as needed alongside deciding what to do. (Your vision is mostly fake; your eyes are rapidly moving around to update it only after you unconsciously pay attention to something.)
> Natural selection weeds out that kind of behaviour.
Not that well since something like half of Americans are myopic…
Anyway, read the paper I linked.
"There is no such thing as a world model, and you don't have one of them."
There is such a thing as a world model in humans, and we all have them otherwise we could not think about or conceptualize or navigate the world. Then you have discussed how to define or construct a useful model or the limitations of a model but that is not relevant to the original point and I'm already aware of that.
It's not some desperate retort. People don't know themselves very well. Look at the research into confabulation, it seems to be standard operating procedure for human brains.
I think if you're going to be strict about this, you have to defend against the point of view that the same 'ding an sich' problem applies to both LLMs and people. And also whether if you had a limit sequence of KL divergences, one from a person's POV of the world, and one from an LLM's POV of texts, what it is about how a person approaches better grasp of reality - and likewise their KL divergence approaches 0, in some sense implying that their world model is becoming the same as the distribution of the world - that can only apply to people.
It seems possible to me that there is probably a great deal of lurking anthropocentrism that humanity is going to start noticing more and more in ourselves in the coming years, probably in both the direction of AI and the direction of other animals as we start to understand both better
It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move (or moves that are better than others) and hence a function that goes from the state of the game to the best move. We just don't have a way to compute this function, but it exists. And that function doesn't need the concept of lookahead, that's just an intuitive way of how could find some of its values. Likewise ML algorithms don't necessarily need lookahead, they can just try to approximate that function with enough precision by exploiting patterns in it. And that's why we can still craft puzzles that some AIs can't solve but humans can, by exploiting edge cases in that function that the ML algorithm didn't notice but are solvable with understanding of the game.
The thing is though, does this really matter if eventually we won't be able to notice the difference?
Is there really a difference between the two? If a certain move shapes the opponent's remaining possible moves into a smaller subset, hasn't AlphaGo "looked ahead"? In other words, when humans strategize and predict what happens in the real world, aren't they doing the same thing?
I suppose you could argue that humans also include additional world models in their planning, but it's not clear to me that these models are missing and impossible for machine learning models to generate during training.
You're confusing the reason why a move is good with how you can find that move. Yeah, a move is good due to how it shapes the opponent remaining moves, and this is also the reasoning we make in order to find that move, but it doesn't mean you can only find that move by doing that reasoning. You could have found that move just by randomly picking one, it's not very probably but it's possible. AIs just try to maximize such probability of picking a good move, meanwhile we try to find a reason a move is good. IMO it doesn't make sense to try to fit the way AI do this into our mental model, since the middle goal is fundamentally different.
The rules of the game are deterministic, but you may be going a step too far with that claim.
Is the game deterministic when your opponent is non-deterministic?
Is there an optimal move for any board state given that various opponents have varying strategies? What may be the best move against one opponent may not be the best move against another opponent.
Therefore the best move cannot be determined by the current board state, as it cannot be determined in isolation from the opponents strategy.
In a perfect information zero sum game, we can theoretically draw a complete game tree, each terminal node ending with a win, loss, or draw. With a full understanding of the game tree we can make moves to minimize our opponent’s best move.
The opponent's moves may be considered non-deterministic, but you can just assume the worst case for you, that is the best case for the opponent, which is the opponent will always play the best move too.
Is it just me or does this read like “here is my assumption about what you said, and now here is my passive aggressive judgement about that assumption”? If you’re not certain about what they mean by the word “understand”, I bet you could ask and they might explain it. Just a suggestion.
LLMs do not have a "planning" module or simulator. There is no way the LLM can plan.
Could build a planning system into an LLM? Possibly and probably, but that is still open research. LeCunn is trying to figure out how to train them effectively. But even an LLM with a planning system does not make it AGI.
Some will argue that iteratively feeding the output embedding back into the input will retain the context but even in those cases it rapidly diverges or as we say "hallucinates"... still happens even with large input context windows. So there is still no planning here and no world model or understanding.
LLMs do not directly model the world; they train on and model what people write about the world. It is an AI model of a computed gestalt human model of the world, rather than a model of the world directly. If you ask it a question, it tells you what it models someone else (a gestalt of human writing) is most likely say. That in turn is strengthened if user interaction accepts it and corrected only if someone tells it something different.
If we were to define that as what "understanding" is, we would equivalently be saying that a human bullshit artist would have expert understanding if only they produced more believable bullshit. (They also just "try to sound like an expert".)
Likewise, I'm not convinced that we can measure its understanding just by identifying inaccuracies or measuring the difference between its answers and expert answers - There would be no difference between bluffing your way through the interview (relying on your interviewer's limitations in how they interrogate you) and acing the interview.
There seems to be a fundamental difference in levels of indirection. Where we "map the territory", LLMs "map the maps of the territory".
It can be an arbitrarily good approximation, and practically very useful, but it's a strong ontological step to say one thing "is" another just because it can be used like it.
This is true. But human brains don't directly model the world either, they form an internal model based on what comes in through their senses. Humans have the advantage of being more "multi-modal," but that doesn't mean that they get more information or better information.
Much of my "modeling of the world" comes from the fact that I've read a lot of text. But of course I haven't read even a tiny fraction of what GPT4 has.
That said, LLMs can already train on images, as GPT4-V does. And the image generators as well do this, it's just a matter of time before the two are fully integrated. Later we'll see a lot more training on video and sound, and it all being integrated into a single model.
Most arguments then descend into confusing the human knowledge embedded in a textbook with the human agency to apply the embedded knowledge. Software that extracts the knowledge from all textbooks has nothing to do with the human agency to use that knowledge.
I love chatGPT4 and had signed up in the first few hours it was released but I actually canceled my subscription yesterday. Part because of the bullshit with the company these past few days but also because it had just become a waste of time the past few months for me. I learned so much this year but I hit a wall that to make any progress I need to read the textbooks on the subjects I am interested in just like I had to this time last year before chatGPT.
We also shouldn't forget that children anthropomorphize toys and dolls quite naturally. It is entirely natural to anthropomorphize a LLM and especially when it is designed to pretend it is typing back a response like a human would. It is not bullshitting you though when it pretends to type back a response about how it doesn't actually understand what it is writing.
I disagree. You can think in terms of a system that doesn't involve predictions at all, but has the same or similar enough outcome.
So an action network just learns patterns. Just like a chess player can learn what positions look good without thinking ahead.
> it will continue to converge towards perfection
Then someone discovered a flaw that made it repeatably beatable by relative amateurs in a way that no human player would be
https://www.vice.com/en/article/v7v5xb/a-human-amateur-beat-...
And
> When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains.
For example, it can reach 25% of accuracy and have an math limit of 26%, so "forever getting closer to 100% with time" would still result in a waste of even infinite resources
I don’t think that’s a given. AlphaZero may have found an extremely high local optimum that isn’t the global optimum.
When playing only against itself, it won’t be able to get out of that local optimum, and when getting closer and closer to it even may ‘forget’ how to play against players that make moves that AplhaGo never would make, and that may be sufficient for a human to beat it (something like that happened with computer chess in the early years, where players would figure out which board positions computers were bad at, and try to get such positions on the board)
I think you have to keep letting it play against other good players (human or computer) that play differently to have it keep improving, and even then, there’s no guarantee it will find a global optimum.
I think to best wrap your head around this stuff, you should look to the commonalities of LLM's, image, generators, and even things like Alpha Zero and how it learned to play Go.
Alpha Zero is kind of the extreme in terms of not imitating anything that humans have done. It learns to play the game simply by playing itself -- and what they found is that there isn't really a limit to how good it can get. There may be some theoretical limit of a "perfect" Go player, or maybe not, but it will continue to converge towards perfection by continuing to train. And it can go far beyond what the best human Go player can ever do. Even though very smart humans have spent their lifetimes deeply studying the game, and Alpha Zero had to learn everything from scratch.
One other thing to take into consideration, is that to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move. And that model is obviously very sophisticated, and if you are going to say that it doesn't understand the game of Go, I would argue that you have a very, oddly restricted definition of the word, understand, and one that isn't particularly useful.
Likewise, with large language models, while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead. As Hinton says, there is no real limit to how sophisticated they can get. When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains. And the closer it gets, the more sophisticated model it needs. In the sense that Alpha Zero needs to "understand" the game of Go to play effectively, the large language model needs to understand "the world" to get better at predicting.