I’m not ignoring how the tech works and this is a simple example. But that doesn’t preclude emergent behavior beyond the statistics.
Did you catch the GPT Othello paper where researchers show, from a transcript of moves, the model learned to model the board state to make its next move? [0]
I’m beginning to think it is reasonable to think of human speech (behavior will come) as a function which these machines are attempting to match. In order to make the best statistically likely response, it should have a model of how different humans speak.
I know GPT is not human, but I also don’t know what form intelligence comes in. I am mostly certain you won’t figure out why we are conscious from studying physics and biochemistry (or equivalently the algorithm of an AI, if we had one). I also believe where ever we find intelligence in the universe, we will find some kind of complex network at its core - and I’m doubtful studying that network we will tell us if that network is “intelligent” or “conscious” in a a scientific way - but perhaps we’d say something about it like - “it has a high attention on apple”.
That said, even playing Othello is still an example of next-token prediction via pattern recognition. Yah, it might be quasi-building a model of sorts, but that's of course just what non-linear predictors do.
Don't get me wrong -- we are also very powerful pattern recognizers.
Why? Any large probabilistic model in your example would also predict "an" due to the high attention on apple.
> One word must come next, but to do a good job modeling what that word will be, wouldn’t you need to consider further ahead than that?
Well, yes, but GPT isn't a human. That's why it needs so much more data than a human to talk so fluently or "reason".