As we don't know for sure what is happening 100% within a neural network, we can say we don't believe that they're thinking and we would still need to define the word thinking. Once LLM's can self-modify, the word "thinking" will be more accurate than it is today.

And when Hinton says at MIT, "I find it very hard to believe that they don't have semantics when they consult problems like you know how I paint the rooms how I get all the rooms in my house to be painted white in two years time," I believe he's commenting on the ability of LLM's to think on some level.

In this case, I think we do if you will check out the paper (https://openaipublic.blob.core.windows.net/neuron-explainer/...). Their method is to

1. Show GPT-4 a GPT-produced text with the activation level of a specific neuron at the time it was producing that part of the text highlighted. They then ask GPT-4 for an explanation of what the neuron is doing.

Text: "...mathematics is done _properly_, it...if it's done _right_. (Take ..."

GPT produces "words and phrases related to performing actions correctly or properly".

2. Based on the explanation, get GPT to guess how strong the neuron activates on a new text.

"Assuming that the neuron activates on words and phrases related to performing actions correctly or properly. GPT-4 guesses how strongly the neuron responds at each token: '...Boot. When done _correctly_, "Secure...'"

3. Compare those predictions to the actual activations of the neuron on the text to generate a score.

So there is no introspection going on.

They say, "We applied our method to all MLP neurons in GPT-2 XL [out of 1.5B?]. We found over 1,000 neurons with explanations that scored at least 0.8, meaning that according to GPT-4 they account for most of the neuron's top-activating behavior." But they also mention, "However, we found that both GPT-4-based and human contractor explanations still score poorly in absolute terms. When looking at neurons, we also found the typical neuron appeared quite polysemantic."

Or maybe the human thought process isn't as sophisticated as we imagined.

I'm not arguing for or against that. It's more the implications of sentience and selfhood implicit in the language many use around LLMs.

What if you ask it to emit the reflexive output, then feed that reflexive output back into the LLM for the conscious answer?

What if you ask it to synthesize multiple internal streams of thought, for an ensemble of interior monologues, then have all those argue with each other using logic and then present a high level answer from that panoply of answers?

What if you do? LLMs don't have reflexive output or internal streams of thought, they are simply (complex) processes that produce streams of tokens based on an inputted stream of tokens. They don't have a special response to tokens that indicate higher-level thinking to humans.

If you direct the model output to itself and don't view it otherwise, how is it not an "internal stream of thought"?

LLMs seem to me to be the "internal streams of thought". I.e. it's not LLMs that are missing an internal process that humans have, but rather it's humans that have an entire process of conscious thinking built on top of something akin to LLM.

I agree completely and I think this is where a lot of people get tripped up. There's no reason to think an AGI needs to be an LLM alone, it might just be a key building block.

Well put, and I agree. My belief is that if a typical person was drugged or otherwise induced to just blurt out their unfiltered thoughts out loud as it crossed their brain, the level of incohesion and false confidence on display would look a lot like an LLM hallucinating.

That's possible I guess but is there positive evidence for that being the case?

The way I phrased it isn't exactly structured to admit any kind of evidence, so let me destructure it. My observation is that:

- The argument that LLMs are missing introspection / inner voice is based on attempting to compare LLMs directly with human minds.

- Human minds have conscious and unconscious parts; for many people, part of the boundary between unconscious and conscious mind manifests as the "inner voice" - the one that makes verbalized thoughts "appear" in their head (or rather perhaps become consciously observed).

- Based entirely on my own experience in using GPT-3.5 and GPT-4, and my own introspection, I feel that GPT-4 bears a lot of resemblance to my inner voice in terms of functioning.

- Therefore I propose that comparing LLMs directly to human minds is unproductive, and it's much more interesting/useful to compare them to the inner voice in human minds: the part of the boundary between unconscious and conscious that uses natural language for I/O.

You seem to have high confidence in how LLMs work.

The text output of a llm is the thought process. In this context the main difference between humans and llms, is that llms can’t have internalized thoughts. There are of course other differences to, like the fact that humans have a wider gamut of input: visuals, sound, input from other bodily functions. And the fact that we have live training.

It's not clear whether or not LLMs have internal thoughts -- each token generation could absolutely have a ton of thought-like modelling in the hidden layers of the network.

What is known is that these internal thoughts get erased each time a new token is generated. That is, it's starting from scratch from the contents of the text each time it generates a word. But you could postulate that similar prompt text leads to similar "thoughts" and/or navigation of the concept web, and therefore the thoughts are continuous in a sense.

True, LLMs definetly has something that is "thought-like".

But todays networks lacks the recursion(feedback where the output can go directly to the input) that is needed for the type of internalized thoughts that humans have. I guess this is one thing you are pointing at by mentioning the continuousnes of the internals of LLMs.

Pride comes before the fall, and the AI comes before humility.

Precisely.

When someone states definitively what LLMs can or cannot do, that is when you know to immediately disregard them as the waffling of uninformed laymen lacking the necessary knowledge foundations (cognitive/neuroscience/philosophy) to even appreciate the uncertainty and finer points under discussion (all the open questions regarding human cognition etc).

They don't know what they don't know and make unfounded assertions as result.

Many would do to refrain from speaking so surely about matters they know nothing about, but that is the internet for you.

https://en.wikipedia.org/wiki/Predictive_coding

> There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text.

To be clear, you're saying that we should just dismiss out-of-hand any possibility that an LM AI might actually be able to explain its reasoning step-by-step?

I find it kind of charming actually how so many humans are just so darn sure that they have their own special kind of cognition that could never be replicated. Not even with 175,000,000,000 calculations for every word generated.

That's a strawman since I didn't argue anything about humans being special. I don't think there is anything necessarily inherently special about human intelligence, I'm just advocating for caution around the language we use to talk about current systems.

All this talk of AGI and sentience and so on is premature and totally unfounded . It's pure sci fi, for now at least.

> I didn't argue anything about humans being special

Above you said about AI LMs:

> There is no "their" and there is no "thought process"

So, unless you're claiming that humans lack a thought process as well, then you're arguing that humans are special.

> All this talk of AGI and sentience and so on is premature and totally unfounded

I don't see any mention of AGI or sentience in this thread?

Also, I don't think anyone could read this transcript with GPT-4 and still claim that it's incapable of a significant degree of self-reflection and metacognition:

https://news.ycombinator.com/item?id=35880148

I reject the words 'self' and 'cognition' in your comment. This is exactly what I'm talking about. A facsimile of them, maybe...

Very true. In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal.

I think LLMs are "Semantic Clouds of Words" + grammar and syntax generator. Someone could just discard the grammar and syntax generator, just use the semantic cloud and create the grammar and syntax by himself.

For example, in writing a legal document, a slightly educated person on the subject, could just use the relevant words put into an empty paper, fill in the blanks of syntax and grammar, alongside with the human reasoning which is far superior than any machine reasoning, till today at least.

The process of editing the GPT* generated documents to fix reasoning is not a negligible task anyway. Sam Altman mentioned that: "the machine has some kind of reasoning", not a human reasoning ability by any means.

My point is, that LLMs are two programs fused into one, "word clouds" and "syntax and grammar", sprinkled with some kind of poor reasoning. Their word clouding ability, is so unbelievable stronger than any human it fills me with awe every time i use it. Everything else is, just whatever!

I think they're much more than that. Or rather, if they're "Semantic Cloud of Words", they're still a hundred thousand dimensional clouds of words, and in those hundred thousand dimensions, any relationship you can think of, no matter how obscure, ends up being reflected as proximity along some subset of dimensions.

Looking at it this way, I honestly wouldn't be surprised if that's exactly how "System 1" (to borrow a term from Kahneman) in our brains works.

What I'm saying is:

> In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal.

It may be much more of a deal than we'd naively think - it seems to me that a lot of what we'd consider "thinking" and "reasoning" can be effectively implemented as proximity search in a high-dimensional enough vector space. In that case, such extracted "Semantic Cloud of Words" may turn out to represent the very structure of reasoning as humans do it - structure implicitly encoded in all the text that was used as training data for the LLMs.

>if they're "Semantic Cloud of Words", they're still a hundred thousand dimensional clouds of words, and in those hundred thousand dimensions, any relationship you can think of, no matter how obscure, ends up being reflected as proximity along some subset of dimensions.

Yes, exactly that. That's what GPT4 is doing, over billions of parameters, and many layers stacked on top of one another.

Let me give you one more tangible example. Suppose Stable Diffusion had two steps of generating images with humans in it. One step, is taking as input an SVG file, with some simple lines which describe the human anatomy, with body position, joints, dots as eyes etc. Something very simple xkcd style. From then on, it generates the full human which corresponds to exactly the input SVG.

Instead of SD being a single model, it could be multimodal, and it should work a lot better in that respect. Every image generator suffers from that problem, human anatomy is very difficult to get right.[1] The same way GPT4 could function as well. Being multimodal instead of a single model, with the two steps discreet from one another.

So, in some use cases, we could generate some semantic clouds, and generate syntax and grammar as a second step. And if we don't care that much about perfect syntax and grammar, we feed it to GPT2, which is much cheaper to run, and much faster. When i used the paid service of GPT3, back in 2020, the Ada model, was the worst one, but it was the cheapest and fastest. And it was fast. I mean instantaneous.

>the very structure of reasoning as humans do it

I don't agree that the machine reasons even close to a human as of today. It will get better of course over time. However in some not so frequent cases, it comes close. Some times, it seems like it, but only superficially i would argue. Upon closer inspection the machine spits out non sense.

[1] Human anatomy, is very difficult to get right, like an artist. Many/all of the artists, point out the fact, that A.I. art doesn't have soul in the pictures. I share the same sentiment.

Clearly there is some process going on to decide what word to pick. Why can’t we call that thinking?

This is it. This comprehension of the chats is symptom of something like linguistic pareidolia. It's an enforced face that is composed of some probabilistic incidents and wistfulness.