> LLMs in general are a poor analog for human intelligence.

I don't think we understand human intelligence nearly well enough to make this claim.

Personally I think that what we consider our "conscious" part is somewhat defined by what we can put into words, and it is in the end putting one word after another.

I think we can conclude a few things that differentiate us from AI. For one, you're capable of formulating multiple thoughts before composing a response to something. It takes time, but you're able to mull over hypotheticals and chase parallel lines of reasoning that AI cannot.

If I asked an AI to respond to this comment, it might write an impassioned defense of itself one-word-after-another, but it wouldn't necessarily base it's defense on freestanding logic. It's foremost goal is writing coherent text, not being right or wrong or minimizing human risk. Paving over any of those assumptions is a fun thought exercise, but doesn't reflect the nature of the technology at-hand or even what we know about human consciousness.

> It takes time, but you're able to mull over hypotheticals and chase parallel lines of reasoning that AI cannot.

How do you know that doesn't happen in some latent space?

> It's foremost goal is writing coherent text, not being right or wrong or minimizing human risk.

That's also true for human bullshitters, it doesn't mean that they're not intelligent.

> How do you know that doesn't happen in some latent space?

If it does, then it contradicts what OpenAI says about their own product. GPT is an autoregressive model trained on text. It's sole purpose, as designed, is to complete text using stochastic reasoning based on the text it learned from. It seems incredibly unlikely (to me) that someone would deliberately mis-design such a system to be capable of independent thought.

If you have evidence of some additional processing layer, I'd love to hear it though.

> That's also true for human bullshitters, it doesn't mean that they're not intelligent.

Bullshit by any other name smells just as bad.

> If it does, then it contradicts what OpenAI says about their own product. GPT is an autoregressive model trained on text. It's sole purpose, as designed, is to complete text using stochastic reasoning based on the text it learned from.

University students complete text using stochastic reasoning based on text they learnt from. Again, that doesn't mean they're not intelligent.

> It seems incredibly unlikely (to me) that someone would deliberately mis-design such a system to be capable of independent thought.

If you train something to produce convincing text then it's perfectly reasonable that it could develop these faculties.

> If you have evidence of some additional processing layer, I'd love to hear it though.

I don't think either of us can conclusively claim that a 175-billion parameter model either does or doesn't do something. I just think there's no particular reason it couldn't.

> Bullshit by any other name smells just as bad.

The question isn't whether GPT produces bullshit or not (it often does), but whether it's fundamentally incapable of thinking like a person. People bullshit too so I don't think that really proves anything.

GPT is fundamentally incapable of thinking like a person. People experience life in a very different way than GPT, and even if we treat all living life as AI-equivalent, GPT would still be a poor approximation of human thought. Even if ChatGPT did develop a personal narrative unbeknownst to it's programmers, that internal dialogue would still be inhuman.

I really don't know what to tell you. You're free to believe whatever you want, but the simplest feasible explanation is the science we used to make these models. At no point was integrating human-like thought a consideration of the model. Just human-like speech.

I think to communicate like a human you need to be able to think like a human, at least to the point where it's indistuinguishable. That's the whole point of the turing test.

The point of the Chinese Room Test is to distinguish between Turing's definition of intelligence and the human definition of intelligence. It poses two separate possibilities; is this AI understanding the language, or simulating the ability to understand the language?

You might find the answer to be arbitrary or meaningless, but it is an important distinction. I could pretend to be a random number generator well enough to fool anyone who's distinguishing me from a computer. That doesn't mean that I'm a reliable source of entropy, or equally as random as the other, similar result. Treated as a black-box they are the same; treated as a mechanical system, they could not be more different.

There's no such thing as a "chinese room test". The point of the thought experiment is that from the point of view of the person outside the room it doesn't matter if the person inside the room speaks chinese or is just following rules. If you're only corresponding with them by text you can't tell the difference. It's exactly the same with large language model and why I don't think statements like "oh it doesn't have intelligence/doesn't "REALLY" understand/it' "only" a large language model" don't really make any sense. As far as I know I could be talking to a large language model right now, but I'm not accusing you of not understanding.

It's like the aliens coming to our planets and saying we can't be intelligent because we're made of meat.

The fact that intelligent and unintelligent text are indistinguishable doesn't prove that ChatGPT's output is intelligent. It more suggests that text is not the right medium for measuring intelligence in the first place. That's what the Chinese room experiment is about.

To me that's a really weird argument. It implies you can't read the theory of relativity and conclude Albert Einstein was a really intelligent guy unless you were in the room with him to make sure he's not a robot in disguise.

This is such a squirrely topic that you presumed that it had goals in this post even though I don't believe you think it does. ;-)

Does it HAVE goals? I don't think there is any evidence that it does. Even if it developed the ability to do all of those things you mentioned, it still wouldn't have any idea what it is doing, let alone why it is doing it, because it's just a language model. That is all it does. And that's why I don't quite understand how people can be so adamant that there is any question about whether it is conscious or not. This whole debate is a trap. It is a brute fact that it is not conscious. It does not have a goal of producing coherent text. That is the goal of the people who wrote it. If we treat the brain as a computer, then GPT lacks the subsystems/algorithms/whatever to do what brains do. It fails at many basic tasks that anything capable of general-purpose reasoning could do. It's often not apparent how spectacularly it can fail because the tasks are so mundane that we don't bother even asking it to do them.

I should have been more clear, because I certainly agree with what you're saying here. However, I think we can pretty clearly define the target output of a model with whatever it was trained on; in this case, GPT is trained on text.

You're right that the model doesn't "know" that necessarily, but it has been designed to produce convincing text when given a small amount of entropy to start with.

What you said is "how we define something is, in the end, putting one word after another".

Well, of course--you just specified what the end is. I'm not trying to be a smart ass. This is just where u have to go to discuss this.

Consciousness isn't a part of us. "Us" is part of consciousness.

> Consciousness isn't a part of us. "Us" is part of consciousness.

I have no idea what that means.

When people make a "conscious decision", or are "conscious" of something, it usually means something they can put into words as opposed to a vague feeling. Being able to communicate with language is what sets us aparts from animals. That's why I think language models are actually really powerful.