News for hackers. Stuff that matters.
Re:The Thoughts the Civilized Keep by jerf on Wednesday September 01, 2021 @04:43PM (#28385298)
No, if you study how GPT-3 works, it becomes pretty clear that that model can't really produce "understanding", even if we can't clearly define what that is very well.

And I don't just mean this from the old point of view of "if we understand it, it's not AI anymore", I mean, the model has serious fundamental limitations. One of them is that it's static, so, once trained, it doesn't learn from its future input. Another is that when it is doing its "what is the highest probability text output to follow this text" (which, itself, is not exactly the sort of question "Strong AI" is the answer to), it by construction can only use a fixed-size window of text. So, for instance, it is by construction simply impossible for GPT-3 to write a novel, or even a short story, because the only way to get it to keep outputting is to provide it a window of the last X KB of output (I don't recall exactly how big it is) for the next continuation, which means that as it is writing the X+5th KB of output, it has literally completely forgotten the first 5KB of output, as completely as if it never existed.

On the flip side, it is clearly very interesting and there is something there. However, I'd submit that what that something is is a something that we don't have an English word for, because up to this point humanity has never encountered anything with a cognitive model like GPT-3. So, it doesn't have "understanding", it has a quality-that-has-no-name (not to be confused with the "quality without a name" which GPT-3 most certainly does not have yet).

Re:The Thoughts the Civilized Keep by TeMPOraL on Wednesday September 01, 2021 @06:21PM (#28386260)
> there is something there

There is indeed. I'll recount my own words after my first proper exposure to the model, in the form of playing custom stories in AI Dungeon:

"I played more of it today, and I can confirm that prolonged exposure makes me feel like I'm half awake, and half in a dream - particularly when the real world interrupts, I feel like I didn't fully wake up.

What's worse, whenever I tried to talk to a real person today, I've felt like I'm just feeding words to an AI, and anticipated "story" to evolve.

Never before has a game messed up with my mind so badly. I'm not sure if it's safe."

To which I got a reply perfectly summarizing the phenomenon:

"AI Dungeon feels a bit like a fever-dream."

--

It's been a year since; thinking back, I feel the trick here is, GPT-3 crosses the threshold of being able to produce plausible and interesting text - where by "interesting" I mean "almost, but not quite, unpredictable". The way other people and their stories are interesting - their words aren't totally random, you have this feeling for the possible trajectories the conversation can take, but you can't quite guess what they're going to say next. GPT-3 does that well enough that it's easy to suspend disbelief and feel as if you're dealing with a mind. But then the feelings that arise in you, that's just you projecting - engaging your mental machinery for dealing with other minds.

Re:The Thoughts the Civilized Keep by jerf on Thursday September 02, 2021 @10:07AM (#28392183)
I did think of clarifying that the window is mostly large enough for AI dungeon to work. Though my understanding is that they do hit it if you go long enough, and the result is, well, exactly what you'd expect from that.

I did play with AI dungeon, and I would say the analogy to a dream is even deeper than it may appear at first, because I found that lucid dreaming techniques basically work on it, which is bizarrely impressive. I would characterize a lot of what one does in lucid dreaming (or, at least, what I do) is basically "expectations management"; if you can learn to expect something to happen in a dream, it frequently does. Not necessarily exactly what you "expected", but it will have a very significant influence. Correspondingly, it is possible (and indeed easy) to basically lead the AI along by using your text prompts to expect things to happen, and in a very dreamlike fashion, while that doesn't mean the exact thing you "expected" will be what the AI produces, it can clearly have very very significant impact on the result.

I should warn you this sort of removes the magic from AI Dungeon as an experience though... I have to consciously "let" it drive the story because it's pretty easy to exert much more control than you're "supposed" to have, when you know how to look behind the curtain.

Re:The Thoughts the Civilized Keep by selfhoster11 on Wednesday September 01, 2021 @04:59PM (#28385452)
> However, I'd submit that what that something is is a something that we don't have an English word for, because up to this point humanity has never encountered anything with a cognitive model like GPT-3. So, it doesn't have "understanding", it has a quality-that-has-no-name (not to be confused with the "quality without a name" which GPT-3 most certainly does not have yet).

I think that terms like "philosophical zombie" may be close to the truth, but I'm guessing some sci-fi work has a term that we could adopt.

Re:The Thoughts the Civilized Keep by balfirevic on Wednesday September 01, 2021 @07:31PM (#28386875)
> No, if you study how GPT-3 works, it becomes pretty clear that that model can't really produce "understanding", even if we can't clearly define what that is very well.

But the quote doesn't say that GPT-3 can't produce understanding because of the specifics of how it works, but "because understanding cannot occur in an isolated computation or behavior, no matter how clever", which is a more general claim.

Re:The Thoughts the Civilized Keep by soco on Thursday September 02, 2021 @06:28AM (#28390424)
And what is "understanding" anyway? Lots of "it can't be this it can't be that" but it doesn't tell what it actually is. Which makes it all easier to slap labels on things which don't actually need labels to just work. Does GPT-3 work? Yes it does and it works without bothering with whatever that "understanding" was. I bet Skynet wouldn't bother either to have an understanding or a soul, it would just... work.
Re:The Thoughts the Civilized Keep by jerf on Thursday September 02, 2021 @09:55AM (#28392026)
I think it would be fair to say that "understanding" requires some sort of internal state change. We can definitely talk about how we did not understand in the past, but now we do. GPT-3 can't experience a state change, by its design.

I don't have to be able nail down exactly what kind of state change "understanding" is to point out that the way we use the word clearly entails one, and GPT-3 clearly can't have one, so therefore it is quite reasonably excluded from "understanding".

(In theory GPT-3 could be extended by continuous online training, but my impression is that convergence is too slow for that to be practical, and it wouldn't look much like "learning".)