> How do we know we’re not doing the same or similar in our brains?

How do we know that we're not made of magic?

That would seem to be the “stochastic parrot” view denounced. But it seems hard to believe. Haven’t you ever had the experience of knowing exactly what you want to say but struggling to find the word for it? How could that be if we’re just randomly selecting words like a souped-up Markov chain?

> Haven’t you ever had the experience of knowing exactly what you want to say but struggling to find the word for it?

I came here to suggest this exact thing. Humans clearly have some understanding of the meaning behind words and sentences. However, I think it's wrong, an overgeneralization, to suggest that ChatGPT is just statistically predicting the next word. While that may technically be true, I think buried deep within it has ways of modeling/encoding common concepts and ideas, maybe similar to how a human models concepts and ideas in their mind.

Then there's the whole problem of consciousness, but I won't get into that here.

>I think buried deep within it has ways of modeling/encoding common concepts and ideas, maybe similar to how a human models concepts and ideas in their mind

Right, maybe meaning and concepts are just “things that usually go together” with different levels of distance.

We’re comparing our minds, which we know very little of, to LLM, which are blackboxes. It seems like a lot of certainty in face of such vast ignorance.

> Haven’t you ever had the experience of knowing exactly what you want to say but struggling to find the word for it? How could that be if we’re just randomly selecting words like a souped-up Markov chain?

All words in the normal cache have a low probability, so you have to hit disk to get the full word list.

We’re not randomly selecting but probabilistically predicting. Haven’t you noticed you predict the next word people will say with very high [________]?

Within a sentence sure. And often arguments are familiar enough that even an essay I’ll know where it’s going. But we do occasionally have original ideas.

I don’t think we know what original really is to make a meaningful distinction between derivative work.

With enough mixing and matching and a tiny bit of randomness, I bet a lot of original thoughts could be created.

These things are hard to quantify but ChatGPT writing does have a certain anodyne character, in my mind, when I read it. Then again, I can't quantify it, I could be biased by my foreknowledge of the writing's provenance, and even if I'm right, the world is full of situations where bland writing is suitable or even desired.

It absolutely does. It’s wishy-washy nature is harder to disagree with, sounds more credible, etc.

Sandwiches!

Indeed, one might conclude that predicting what to do in the next moment (analogous to what to write next in the case of ChatGPT, which can only write) is all we do as humans. At every moment of the day, we are taking all our past experiences and the present as inputs, and predicting what we should do next based on that.

This is in fact the whole premise of Jeff Hawkins' book "On intelligence". Prediction is at the core of how we learn/act. Since we are born, we are just predicting everything that will happen and our brain learns when we see/hear/sense something that we can't predict.

I love the philosophy behind Gustafson's Law: the problem size increases in response to computational resources.

You get a simple computer, you can drop bombs more accurately. Then you make word processors. Then text based simulation aka games. Then turn books into digital format. Then whole online libraries, ecommerce, social media, cryptocurrency, and so on.

Seeing a LLM as purely language based might be missing the forest for the atoms.

A token generator corresponds to reflexes: a fuzzy hash table of situations and responses. Cats and dogs have that.

Assigning meanings to tokens and establishing relationships between them is where intelligence begins. GPT continues sequence CGGA with T because there is a similar sequence in its dataset, but we interpret T as a tree and see how this meaning relates to A. I believe GPT will solve this problem to some extent in 10 years and will finally earn its real AI badge.

Above it there is abstract mind that only top scientists use to some extent.

I won't speak for what's "all the way down", but on whatever level humans perform reflective reasoning, it's self-evident that we don't simply "predict the next word" in the way a LLM does.

Whether or not a LLM is capable of "creating anything new" is a different argument altogether. "New" is in eye of the beholder and it is often said that nothing is new under the sun, but we already know that unthinking processes within nature are capable of producing beauty far and beyond anything that even the greatest human artist could ever hope to accomplish. That a large LLM is unthinking doesn't preclude it from producing art.

Please provide evidence that intelligent speech isn’t simply predicting the next word, as it’s obviously not self-evident to me.

I'm not sure it counts as "evidence" because I can't show it to you, but based on personal introspection, whenever I write something it starts with a barely-linguistic thought I want to communicate that gradually gets expanded into as much prose as is necessary to get across the concept whilst establishing context and defending from the most probable detractors.

I literally never just start writing and then just keep adding words.

It’s not evidence, sure, but what you describe, the “feel of it” is most of what we have to work with at the moment, unfortunately.

But it clearly is not self-evident that intelligence is or isn’t prediction.

Nothing is self-evident in a field we know so very little about. That’s my main objection, we should be humble and curious, not dogmatic.

It's pretty much self-evident when you worked with 3-6 that this isn't how it work. The idea come before the vocalization.

Or even when you talk in a foreign language you're semi-competent with.

I’ve spent many years with infants and learned 3 languages aside from my mother tongue. I don’t find it self-evident at all.

You’re gonna need to elaborate why you think so if you’re willing to convince someone.

That the idea form before it's vocalized?

A three year old reaching toward any toy, then saying 'mine!'.

Stumbling on a specific word, stopping your sentence, then proceed to replace it with a combination of words.

Or any time you paraphrase basically.

Not true in my experience. To me, words come first, the idea is a consequence I realize after a few of them have been stringed together.

Knowing where the words come from, how “the brain” knows the next one before my conscious self does has always been a mystery to me. I’ve heard Sam Harris describing a similar feeling in his podcast.

You never paraphrased something despite knowing the exact word?

[edit] Or explained high level CS concepts to an undergrad at your school?

Sure, but how's that indicative that prediction isn't taking place?

I have the idea before i have thought of the word i would use to express it. Sometimes, i don't even have word to express it, "you know what i mean?".

And like i said. Children will often act, then talk. Or seek their words for long period of time, and when you propose a word to help them vocalize, they can accept it or reject it.

I have another example. Do you ever have slip of the tongue? You're talking about "yoga", tell yourself while speaking "i really shouldn't say 'yoda' here", and say the word "yoda" anyway? This happened to me a lot when i was younger (my speed of thought is lower those days, i don't think i slipped this way in two or three years). I still think a bit faster than i speak tbh.

All of that happens to me as well. I just don’t see how it couldn’t be explained by prediction from probability distribution.

The fact that you can’t find a word could be that you’re stuck in a Markov chain wall or loop:

https://news.ycombinator.com/item?id=34688673

The thing is, we just don’t know how any of that work. Our speculation is as good as any.

I've always hated the idea promoted by some AI researchers that "intelligence is just prediction." Specifically in the context of your challenge, since a brain must decide the next word before uttering it, is it not vacuously true that any conceivable method of producing speech is "predicting" it? You're asking for evidence that it isn't; I'm asking for evidence that question has any meaning.

Sure, any production in that case can be classified as prediction. What I’m saying is that such distinction, if at all possible, is far from self-evident.

Anytime you say “uh, what’s that word again?”

So you’re thinking of a concept, you can picture it and it’s impact on your environment, but can’t quite remember the word for it.

Or, you reached a point in the Markov chain where none of the next guesses rank high enough if the whole paragraph is taken into account. It often feels like that, in fact, you can’t figure it out until you take another “route”.

Of course, my anecdote is just as valid as yours. My point is we have no freaking idea what’s going on and shouldn’t jump to conclusions.

The content of intelligent speech is not driven by statistical relationships between the spoken lexemes, this is self-evident.

For example, let's say you're planning to approach your boss about receiving a raise, the contents of your speech will be driven by a personal theory of mind regarding how your boss perceives you, your own memory of who you are and how you like to be perceived, the cadence and tenor of speech you imagine your boss is receptive to, information contextually relevant to the purpose of the speech like your proven history of success within the organization, as well as a hundred other variables driven by physiological factors, culture, social habituation and more. "Predict the next word" just doesn't describe intelligent speech in any meaningful way.

So, you thrown in theory of mind, physiological factors and culture without defining what that means or showing that they don't all derive from the same underlying statistical relationships in order to make your argument more robust? That's not how it works.

Defining intelligence is fundamental philosophical problem, a discussion that's thousands of years old in the making. To call something in this field self-evident is either ignorance or self denial.

Someone deaf and blind still form thoughts but don’t output them in words. Words follow cognition, it’s sometimes obvious when you have a clear concept in mind but forget the word.

Edit: the thought is maybe a prediction, but language seems not to be the abstraction it’s operated on.

Of course they do. They still have language. How else would they interact with the world?

You're asking for the evidence of a non existing factor in a complex process not fully understood... I'd say it's on you to provide it.

Why is predicting the next word proof of intelligent speech?

That’s not what I’m asking.

The grandparent said it’s self-evident that humans do not simply predict the next word. I asked for evidence of that self-evidence.

This. I think the biggest distinction between biological neural nets that seem to be self aware and those that are not is simply metacognition. I suspect the actual stimuli responses (decision making) is virtually the same but self awareness adds a layer after the fact of reasoning about the decision.

IF I'm right, one could reasonably argue that metacognition is simply a tool to help adjust the stimuli response for future decisions. One that has become complex enough that it rose to the level to give the sense of self awareness. A sort of back propagation tool to put it in ML terms.

Further, if a system isn't adjusting weights and biases as it goes, and metacognition/self awareness is just a mechanism to do so, then we're not so different from these models aside from them being orders of magnitude simpler.

We like to think the way we think is somehow magic and discredit these systems as a way of keeping our thinking elevated. Maybe our brains are not that special just far more advanced than our current technological capabilities.

Maybe I'm way off base, but like you said, it's dangerous to just assume these systems are missing some magic and therefore just bs generators (as if people aren't pretty consistent bs generators)

tl;dr: Agreed. Maybe all biological neural networks are untrustworthy bs generators too. A pile of bs built on previous bs all the way down to the beginning of life.

>What if it’s all just a “computational guessing game” all the way down

then we would be categorically unable to meaningfully work on purely analytical problems that have nothing to do with induction, which is how these learning systems work. Human beings can invent mathematical proofs and build machines that do symbolic computation, ChatGPT can't. At best it can try to copy a solution that humans have written down somewhere or mangle it in the process.

> then we would be categorically unable to meaningfully work on purely analytical problems

How so? Sorry, but to categorically affirm such bold claim you need more substance.

This is a hard problem, perhaps the hard problem, and we’ve been banging our heads for millennia with very little progress.

Now we create a machine that pretty much passes the Turing test. It’s certainly feels more human than some telemarketing call I’ve received. And we’re not going to investigate it further but rather dismiss it as a cool trick? For all we know, that’s all intelligence may be.

>How so? Sorry, but to categorically affirm such bold claim you need more substance.

It's not a bold claim, it's a trivial claim. No existing learning system can reason. If it could we could hand it axioms and it could do the same deductive reasoning that you can. By definition and how these systems work, all they do is generalize from particular examples, and you can't do maths by empirically looking for numbers in a set of data, because there's an infinite amount of them. That's why ChatGPT spits out nonsense if you give it two large numbers two multiply, but your TI-83 does it perfectly on a battery. We can program deterministic, deductive methods into machines, we have no machine that develops them.

Now is there some architecture that can at some point learn how to do more than statistical prediction? Sure but this isn't it. Passing the Turing test and making people feel excited has little to do with what actual intelligence a system has. The Mechanical Turk in the 18th century fooled people, but it wasn't a intelligent machine, it was an elaborate contraption that mimics human behavior. And people understandably conflate the latter with the former.

I don’t consider ChatGPT to be self aware. I think most people don’t. No one, however, can pinpoint exactly why.

The vast majority of the population can’t deduce anything from axioms or even multiply large numbers. Yet most won’t doubt their self awareness.

If you’re using “self-evident” and “trivial” when discussing AI, conscious, intelligence, you’re fooling yourself.

Picture a line—say, the line defined by y=2x + 3.

Now, tell me what's contained inside that line. Not "what does it mean" or "what's it made up of"; "what is contained inside it?"

The question doesn't make sense. There's no "inside" of a line. It's a one-dimensional mathematical construct. It fundamentally cannot "contain" anything.

"What's inside the line?" is a similar question to "Is ChatGPT self-aware?"—or, more aptly, to "What is ChatGPT thinking?" It's a static mathematical construct, and thinking is an active process. ChatGPT fundamentally cannot be said to be thinking, experiencing, or doing any of the other things that would be prerequisites for self-awareness, however you want to define that admittedly somewhat nebulous concept. Thus, to even ask the question "Why don't you think ChatGPT is self-aware?" doesn't make sense. It's not that far different from asking asking "Why don't you think your keyboard/pencil/coffee mug is self-aware?"

The intelligence of all humans is roughly analogous in ability—even if a given human has not learned to do formal logical deduction and inference, the fundamental structure and processing of the human brain is unquestionably capable of it, and most humans do so informally with no training at all.

Attempting to cast doubt on the human ability to reason, to comprehend, and to synthesize information beyond mere stochastic prediction reflects a very naïve, surface-level view of humans and cognition, and one that has no grounding in modern psychology or neuroscience. Your continued insistence, through several sub-threads, that we cannot be sure we are any better than ChatGPT is very much an extraordinary claim, and you have provided no evidence to support it beyond "I can't imagine a proof that we are not."

Maybe go do some research on how our brains actually work, and then come back and tell us if you still think we're all just predictive chatbots.

Haha, yeah. OK internet stranger. Take a deep breath and perhaps consider why questioning the certainties you so dearly hold throws you into an ad hominem fallacy.

Maybe check if the consensus on neuroscience is that the brain is definitely-for-sure-certainly not a predictive machine while you’re at it.

I work for a psychology and neuroscience department, and have done for over a decade now.

Does that give me genuine academic credentials? Pff, no.

Does it mean I have a reasonable high-level grounding in modern understanding of how the brain works? Yes, it does.

Again: You have made an extraordinary claim. You need to provide extraordinary evidence to support it, not just toss accusations of ad-hominem attacks at anyone who points out how thin your argument is.

I have claimed nothing, only cast doubt on the baseless certainties that have been floating around this subject.

To summarize what I'm trying to convey:

- (Variations of the original claim) ChatGPT works nothing like the brain, it's just a prediction machine

- (Me all over this thread) Do we know how the brain works? Do we know what properties may emerge from a blackbox LLM?

If you think questioning awareness is extraordinary (or new) I advise you to read Descartes, the Boltzmann brain thought experiment and epistemology in general.

PS: you replied with credentials rather than arguments, which is still on the logical fallacy ground.

I'm very much just predicting the next word when I talk. I don't know the grammar of the sentence and everything I will say when I start the sentence. I just start talking and try to hit the right notes and, if its a topic I've covered before, have some vague plan for the whole thing. Maybe other people's experiences are different, but when it comes to my natural language only and not my ability for reasoning or abstract thought - I am absolutely not doing anything significantly more complicated than chatGPT is doing.

A human mind probably is essentially a prediction engine, but it is predicting everything one can perceive, not just language. That's how a human can throw and catch a ball, or walk over uneven ground, or know that Mom will be mad about the mess.

Our intelligence is trained by hard reality, and so our use of language fits into a much broader context. "Touching a hot stove" is a metaphor because we know viscerally what would happen if we actually did it.

The limitation of ChatGPT is not that it predicts, the limitation is the incredible paucity of data from which it tries to predict. Language encodes some structure of the world because that's where it was created. But to the extent that can be inferred by ChatGPT, it's a weak and unreliable echo of the original reality.

ChatGPT exists in a featureless, timeless void with only patterns of characters in its memory. It can't tell real from fake (from our perspective) because ALL of its entire reality is just those patterns of text. The words are all real, and all there is.

This doesn't even get into why human minds make predictions. ChatGPT makes predictions because we built it to, and we prompt it to. Why do our brains? The trite answer is survival of the fittest, but that just raises the question of why we want to survive.

The point is, there is a fundamental difference between a living intelligence, and a machine learning tool. ChatGPT will sit quietly until prompted, and then answer only what it is asked. A human won't. I'm not sure that comes down to intelligence, or something else. Even bugs don't sit still and do what they're asked.

How well GPTZero can detect ChatGPT generated text by measuring perplexity (how random the words choices are) and burstiness (how diverse the sentence structure is) shows that whatever algorithm our brain uses has stronger creative capabilities than this LLM.

GPT-3.5 isn’t a great writer like AlphaGo is a great go player. Maybe one day AI will generate better scripts and novels than humans, but not this model.

Medium-quality writing is ok for informative content though, but it’s problematic when the model doesn’t know fact from fiction. That’s the important complaint.

Is it dangerous? Maybe.

But is it useful? Not if it’s wrong too often.

You’re right that this tech should be taken seriously, but so should the hallucination problems. These problems can be solved. And maybe they should be solved before anyone trusts it with serious questions.

> it’s problematic when the model doesn’t know fact from fiction.

This is in no way unique to an AI. Have you ever interacted with humans? Half the population thinks the other half can't tell fact from fiction. The other half thinks the same about the first half. We're all wrong about all the time.

There's a fundamental difference between "some people are wrong some of the time"—or even "half the population has trouble telling fact from fiction some of the time", if we grant that as true for the sake of argument—and "ChatGPT (and similar ML algorithms) don't even have a metric to determine truth from fiction; they just predict what's the most likely set of words to stick together in response to your prompt."

ChatGPT fundamentally cannot ever know when it's wrong. I should hope it goes without saying that that's not true of humans.

From what I’ve read, LLM can’t reliably detect itself.

You are just a bullshit generator - and so am I!

>> How do we know we’re not doing the same or similar in our brains?

Because if I ask you to calculate 85 + 73 in your head, you will not start searching for the string that completes "85 + 73 = " with maximum probability, but instead try to add those two numbers using an algorithm you learned at school.

Which means you can compute stuff, just like a computer, and not only guess stuff, just like an LLM.

You’re just repeating that computing is not predicting without explaining why.

Also, LLMs are not grep. If the string doesn’t match, it will come up with something it deems plausible, like us. It’s even bad at math in eerily similar ways to us, like off-by-a-decimal-place errors.

>> You’re just repeating that computing is not predicting without explaining why.

Yes, because I assume you're a programmer, or a computer scientist, and you already understand the difference.

To clarify, the difference is that a program that computes the most likely next token in a sequence, like a language model (LM) is one kind of program, whereas the set of all program contains infinitely many programs that are not LLMs; and an LLM cannot calculate every such program, nor can it perform the same calculations as those programs, it can only calculate the most likely next token in a sequence.

Is the distinction clearer now? Also, are you a programmer? If so, I would point to the wikipedia article on Turing Machines or the one on the theory of computation. It's perhaps not a very easy subject to explain in a HN comment, but if you have some background in programming computers you should be able to grok it. You will see for example that a Turing machine is a device that manipulates symbols to produce other symbols, which is what we generally mean by "computation" and not a device that predicts the next token in a sequence, as an LLM does.

Sorry if I assume too much about your background - this is HN after all.

No worries, you were right on your assumption, I am a programmer, like you seem to be as well.

> like a language model (LM) is one kind of program, whereas the set of all program contains infinitely many programs that are not LLMs

I don’t know if it’s been formally verified, but it’s pretty safe to bet that ChatGPT is Turing Complete. If so, than your statement is false. ChatGPT can emulate every computable problem.

>> No worries, you were right on your assumption, I am a programmer, like you seem to be as well.

Of course :)

>> I don’t know if it’s been formally verified, but it’s pretty safe to bet that ChatGPT is Turing Complete. If so, than your statement is false. ChatGPT can emulate every computable problem.

I think that's unlikely. For Universal Turing Machine expressivity, a system needs to have something to function as an infinite Turing tape, and where's that in an LLM? ChatGPT famously has a limited input buffer and it doesn't even have a memory, as such (it forgets everything you tell it, hence why a user's input and its own answers have to be fed back to it continuously during a conversation or it loses the thread).

Besides, OpenAI themselves, while they have made some curious claims in the past (about GPT-3 calculating arithmetic) seem to have backtracked recently and nowadays if you ask ChatGPT to calculate the result of a computation it replies with a sort-of-canned reply that says it's not a computer and can't compute. Sorry I don't have a good example, I've seen a few examples of this with Python programs and bash scripts.

Anyway you could easily test whether ChatGPT (or any LLM) is computing: ask it to perform an expensive computation. For example, ask it to compute the Ackermann function for inputs 10,20, which should take it a few hundred years if it's actually performing a computation. Or ask it to generate the first 1 million digits of pi. It will probably come up with a nonsense answer or with one of it's "canned" answers, so it should be obvious it's not computing the result of the computation you asked it to.

Btw, I think one could argue that a Transformer architecture is Turing-complete, in the sense that it could be, in principle, trained to simulate a UTM; I seem to remember there are similar results for Recurrent Neural Networks (which are, however, a different architecture). But a Transformer trained to generate text is trained to generate text, not to simulate a UTM.

> infinite Turing tape

That's the ideal machine Turing describes, but it's a very narrow definition we don't really use. If that were the case, nothing would ever be Turing-complete, not even the computer we are writing this on. If we can consider a cellular automaton and even the single x86's MOV instruction Turing-complete, I'm pretty sure ChatGPT will qualify.

>it doesn't even have a memory, as such (it forgets everything you tell it…

It really doesn't. Have you tried it? You can bring back context from several prompts before. One of the remarkable things about it, in fact.

>Btw, I think one could argue that a Transformer architecture is Turing-complete

Bingo. And you can probably tweak your prompt to steer it. The things people at Reddit have been able to persuade ChatGTP to do against its directives is tantalizing.

For a UTM, you need something to function as an infinite Turing tape. For example a mechanism that can loop arbitrarily. On a machine with finite resources, running an infinite program will exhaust the machine's resources. But ChatGPT has nothing like that.

As to ChatGPT's memory that's not about UTM expressivity, but another reason why it can't be any kind of computing device.

ChatGPT and other language models of its kind don't have memories. It can retrieve context from earlier prompts because each of your prompts, and the language model's responses to your prompts, are prepended to your latest prompt and then the whole thing passed to the model. It doesn't remember what you said earlier, it just reads it again each time.

Finally, a neural net architecture and a trained model are two different things. For example, you could train one Convolutional neural net to recognise cats, and another to recognise dogs. Suppose you could train a Transformer to simulate a UTM. ChatGPT is not trained to simulate a UTM, it's trained to predict text. So it can't simulate a UTM.

But, after all this discussion I have to ask: why do you say that "it’s pretty safe to bet that ChatGPT is Turing Complete"? What does ChatGPT have in common with cellular automata, for instance?

When I don't know how a stage magician's trick works, I don't immediately default to assuming that they must have occult powers.