If the Gödel incompleteness theorem applies here, then the explanations are likely … incomplete or self-referential.

What leads you to suspect that Gödel incompleteness may be relevant here?

There's no formal axiom system being dealt with here, afaict?

Do you just generally mean "there may be some kind of self-reference, which may lead to some kind of liar-paradox-related issues"?

I commented in another answer but you can consult https://etc.cuit.columbia.edu/news/basics-language-modeling-...

Some training forms include entailment : “if A then B”. I hope this is first order logic which does have an axiom system :)

The relevance is because all (all known buildable aka algorithmic, and sufficiently powerful) models of computation are equivalent in terms of formal computability, so if you could violate/bypass the Godel or Turing theorems in neural networks, then you could do it in a Turing machine, and vice versa. (That's my understanding, feel free to correct me if I'm mistaken)

Well... , yeah, but, these models already produce errors for other unrelated reasons, and like...

Well, what exactly would we be showing that these models can’t do? Quines exist, so there’s no general principle preventing reflection in general. We can certainly write poems (etc.) which describe their own composition. A computer can store specifications (and circuit diagrams, chip designs, etc.) for all its parts, and interactively describe how they all work.

If we are just saying “ML models can’t solve the halting problem”, then ok, duh. If we want to say “they don’t prove their own consistency” then also duh, they aren’t formal systems in a sense where “are they consistent (as a formal system)?” even makes sense as a question.

I don’t see a reason why either Gödel or Turing’s results would be any obstacle for some mechanism modeling/describing how it works. They do pose limits on how well they can describe “what they will do” in a sense of like, “what will it ‘eventually’ do, on any arbitrary topic”. But as for something describing how it itself works, there appears to be no issue.

If the task to give it was something like “is there any input which you could be given which would result in an output such that P(input,output)” for arbitrary P, then yeah I would expect such diagonalization problems to pop-up.

But a system having a kind of introspection about how it works, rather than answering arbitrary questions about its final outputs (such as, program output, or whether a statement has a proof), seems totally fine.

Side note: One funny thing: (aiui) it is theoretically possible for a oracle that can have random behavior, to act (in a certain sense) as a halting-oracle for Turing machines with access to the same oracle.

That’s not to say that we can irl construct such a thing, as we can’t even make a halting oracle for normal Turing machines. But, if you add in some random behavior for the oracles, you can kinda evade the problems that come from the diagonalization.

I don't have a rigorous argument but as I recall Rice's theorem says that computers cannot compute nontrivial properties about programs, in general. So we cannot have virus checkers, etc. (Well we actually do have imperfect virus checkers, of course). So following that, a nontrivial property such as self-explanation, e.g. correctly explain what a neural network is doing, for all possible inputs. (But again this is not fully clear to me, someone might point out that for a specific program this is different and Rice's theorem is irrelevant.)

The Goedel Incompleteness Theorem has no straightforward application to this question.

It would if the language model did reasoning according rules of logic. But they don't. They use Markov chains.

To me it makes no sense to say that a LLM could explain its own reasoning if it does no (logical) reasoning at all. It might be able to explain how the neural network calculates its results. But there are no logical reasoning steps in there that could be explained, are there?

Honest question: are we sure that it doesn’t do logical reasoning?

IANAE but although an LLM meets the definition of a Markov Chain as I understand it (current state in, probabilities of next states out), the big black box that spits out the probabilities could be doing anything.

Is it fundamentally impossible for reasoning to be an emergent property of an LLM, in a similar way to a brain? They can certainly do a good impression of logical reasoning- better than some humans in some cases?

Just because an LLM can be described as a Markov Chain doesn’t mean it _uses_ Markov Chains? An LLM is very different to the normal examples of Markov Chains I’m familiar with.

Or am I missing something?

In any case, coemu is an interesting related idea to constrain AIs to thinking in ways we can understand better:

https://futureoflife.org/podcast/connor-leahy-on-agi-and-cog...

https://www.alignmentforum.org/posts/ngEvKav9w57XrGQnb/cogni...

My understanding is that at least one form of training in the RLHF involves supplying antecedent and consequent training pairs for entailment queries.

The LLM seems to be only one of the many building blocks and is used to supply priors / transition probabilities that are used elsewhere in downstream part of the model.

All programs that you can fit on a computer can be described by a sufficiently large Markov chain(if you imagine all the possible states the memory as nodes) Whatever the human brain is doing is also describable as a massive Markov chain.

But since the markov chain becomes exponentially larger whit the amount of states this is a very nitpicky and meaningless point.

Clearly to say something its a markov chain and have that mean something you need to say the thing its doing could be more or less compressed to a simple markov chain for bigrams or something like that, but that is just not true empirically, not even for gpt2. Just this is already pretty hard to make into a reasonable size markov chain https://arxiv.org/abs/2211.00593.

Just saying that it outputs probabilities from each state is not enough, the states are english strings, there's (number of tokens)^contex_lenght possible states for a certain length that's not a reasonable markov chain that you could actually implement or run.

> Honest question: are we sure that it doesn’t do logical reasoning?

It's not the Creature from the Lagoon, its an engineering artifact created by engineers. I haven't heard them say it does logical deduction according to any set of logic-rules. What I've read is it uses Markov chains. That makes sense because basically an LLM given a string-input should reply with another string that is the most likely follow-up string to the first string, based on all the texts it crawled up from the internet.

If internet had lots and lots of logical reasoning statements then a LLM might be good at producing what looks like logical reasoning, but that would still be just response with the most likely follow-up string.

The reason the results of LLMs are so impressive is that at some point the quantity of the data makes a seemingly qualitative difference. It's like if you have 3 images and show them each to me one after the other I will say I saw 3 images. But if you show me thousands of images 24 per second and the images are small variations of the previous images then I say I see a MOVING PICTURE. At some point quantity becomes quality.

That's probably one of the reasons why you'd use GPT-4 to explain GPT-2.

Of course, if you were trying to use GPT-4 to explain GPT-4 then I think the Gödel incompleteness theorem would be more relevant, and even then I'm not so sure.

So is the word "word" but that seems to have worked out OK so far. I can explain the meaning of "meaning" and that seems to work OK too. Being self-referential sounds a lot more like a feature than a bug. Given that the neurons in our own heads are connected to each other and not any ground truth, I think LLMs should do just fine.

As long as lazy evaluation exists, self-reference is fine, no?

Hofstadter talks about something similar in his books.

Are there any examples of an LLM developing concepts that do not exist or cannot be inferred from its training set?

It is by definition impossible for an LLM to develop a concept that 'cannot be inferred from its training set'.

On the other hand, that is an incredibly high bar.

"Cannot be inferred from its training set" is a pretty difficult hurdle. Human beings can infer patterns that aren't there, and we typically call those hallucinations or even psychoses. On the other hand, some unconfirmed, novel patterns that humans infer actually represent groundbreaking discoveries, like for example much of the work of Ramanujan.

In a real sense, all of the future discoveries of mathematics already exist in the "training set" of our present understanding, we just haven't thought it all the way through yet. If we discover something new, can we say that the concept didn't exist, or that it "couldn't be inferred" from previous work?

I think the same would apply to LLMs and their understanding of the way we encode information using language. Given their radically different approach to understanding the same medium, they are well poised to both confirm many things we understand intuitively as well as expose the shortcomings of our human-centric model of understanding.

The training sets are so poorly curated we will never know...

Tautologically, every concept that anything (LLM, or human, or alien) develops can be inferred from the input data(e.g. training set), because it was.

No, it wasn't, language itself didn't even exist at one point. It wasn't inferred from training data into existence because such examples existed before. Now we have a dictionary of tens of thousands of words, which describe high level ideas, abstractions, and concepts that someone, somewhere along the line had to invent.

And I'm not talking about imitation nor am I interested in semantic games, I'm talking about raw inventiveness. Not a stochastic parrot looping through a large corpus of information and a table of weights on word pairings.

Has AI ever managed to learn something humans didn't already know? It's got all the physics text books in its data set. Can it make novel inferences from that? How about in math?

> No, it wasn't, language itself didn't even exist at one point.

Language took dozens of millennia to form, and animals have long had vocalizations. Seems like a natural building on top of existing features.

> Has AI ever managed to learn something humans didn't already know?

AlphaZero invented all new categories of strategy for games like Go, when previously we thought almost all possible tactics had been discovered. AIs are finding new kinds of proteins we never thought about, which will blow up the fields of medicine and disease in a few years once the first trials are completed.

> Language took dozens of millennia to form

Sure, but in a simulated evolutionary algorithm, you can hit a few hundred generations in a matter of seconds.

Indeed, the identification of an abstraction, followed by a definition of that abstraction and an enshrinement of the concept in the form of a word or phrase, in and of itself, shortcuts the evolutionary path altogether. AI isn't starting from scratch: it's starting from a dictionary larger than any human alive knows and in-memory examples of humans conversing on nearly every topic imaginable.

We never thought "all possible tactics" had been discovered with Go. We quite literally understood that Go had a more complex search space than Chess, with far more possible moves and outcomes. And I don't think anyone has any kind of serious theorem that "all possible tactics" have been discovered in either game, to this day.

That being said, Go and Chess are enumerable games with deterministic, bounded complexity and state space.

The protein folding example is a neat one, I definitely think it's interesting to see what develops there. However, protein folding has been modeled by Markov State models for decades. The AlphaFold breakthrough is fantastic, but it was already known how to generate models from protein structures: it was just computationally expensive.

It was also carefully crafted by humans to achieve what it did: https://www.youtube.com/watch?v=gg7WjuFs8F4. So this is an example of humans using neural network technology that humans invented to achieve a desired solution to a known problem that they themselves conceived. The AI didn't tell us something we didn't already know. It was an expert system built with teams of researchers in the loop the whole way through.

Coming up with new language is rarely ever coming up with new concepts that didn't exist until the word. We come up with high-level abstractions because there already exists a material system to be described and modeled. Language that doesn't describe anything that doesn't already exist is more like babbling.

Not really, no. There are plenty of intangible abstractions that don't describe material systems. Take imaginary numbers, for example. Or the concept of infinity or even zero, neither of which exists in the physical world.

The reason why "naming things" is the other hard problem in computer science, after cache invalidation, is that the process of identifying, creating, and describing ideal abstractions is itself inherently difficult.

I'm really curious what kind of concept you might have in mind. Can you give any example of a concept that if an LLM developed that concept then it would meet your criteria? It might sound like a sarcastic question but it's hard to agree on the meanings of "concepts that do not exist" or "concepts that cannot be inferred" maybe you can give some examples.

EDIT: I see below you gave some examples, like invention of language before it existed, and new theorems in math that presumably would be of interest to mathematicians. Those ones are fair enough in my opinion. The AI isn't quite good enough for those ones I think, but I also think newer versions trained with only more CPU/GPU and more parameters and more data could be 'AI scientists' that will make these kinds of concepts.

First of all, our own explanations about ourselves and our behaviour are mostly lies, fabrications, hallucinations, faulty re-memorization, post hoc reasoning:

"In one well-known experiment, a split-brain patient’s left hemisphere was shown a picture of a chicken claw and his right hemisphere was shown a picture of a snow scene. The patient was asked to point to a card that was associated with the picture he just saw. With his left hand (controlled by his right hemisphere) he selected a shovel, which matched the snow scene. With his right hand (controlled by his left hemisphere) he selected a chicken, which matched the chicken claw. Next, the experimenter asked the patient why he selected each item. One would expect the speaking left hemisphere to explain why it chose the chicken but not why it chose the shovel, since the left hemisphere did not have access to information about the snow scene. Instead, the patient’s speaking left hemisphere replied, “Oh, that’s simple. The chicken claw goes with the chicken and you need a shovel to clean out the chicken shed”" [1]. Also [2] has an interesting hypothesis on split-brains: not two agents, but two streams of perception.

[1] 2014, "Divergent hemispheric reasoning strategies: reducing uncertainty versus resolving inconsistency", https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522

[2] 2017, "The Split-Brain phenomenon revisited: A single conscious agent with split perception", https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf

that is absolutely fascinating and also makes me extremely uncomfortable

Neurology is full of very uncomfortable facts. Here's one for you: there are patients who believe their arm is gone even though it's still there. When the doctor asks whose arm that is, they reply it must be someone else's. The brain can simply refuse to know something, and will adopt whatever delusions and contortions are necessary. Which of course leads to the realization that there could be things we're all incapable of knowing. There could be things right in front of our faces we simply refuse to perceive and we'd never know it.

Oliver Sacks' A Leg To Stand On is a lengthy discussion of that, including his own experiences after breaking a leg---IIRC, at one point after surgery but before he starts physical therapy, he wakes up convinced that a medical student has played a prank by removing his leg and attaching one from a cadaver, or at least sticking a cadaver's leg under his blanket. (ISTR he tries to throw it out of bed and ends up on the floor.)

Famously, our nose is literally right in front of our faces and the brain simply "post-processes" it out of the view.

After breaking my arm, split in two, pinching the nerve and making me unable to move it for about a year, I still feel as if the arm is "someone else's", as if I am moving an object in VR, not something which is "me" or "mine".

This is why I suggest that curious individuals try a hallucinogen at least once*. It really makes the fragility of our perception, and how it’s held up mostly by itself, very apparent.

* in a safe setting with support, of course.

I can confirm lucid dreaming does the same.

It's probably way worse than we can imagine.

Reading/listening to someone like Robert Sapolsky [1] makes me laugh I could have ever hallucinated about such a muddy, not even wrong concept as "free will".

Furthermore, between the brain and, say, the liver there is only a difference of speed/data integrity inasmuch as one cares to look for information processing as basal cognition: neurons firing in the brain, voltage-gated ion channels and gap junctions controlling bioelectrical gradients in the liver, and almost everywhere in the body. Why does only the brain has a "feels like" sensation? The liver may have one as well, but the brain being an autarchic dictator perhaps suppresses the feeling of the liver, it certainly abstracts away the thousands of highly specialized decisions the liver takes each second solving adequately the complex problem space of blood processing. Perhaps Thomas Nagel shouldn't have asked "What Is It Like to Be a Bat?" [2] but what is it like to be a liver.

[1] "Robert Sapolsky: Justice and morality in the absence of free will", https://www.youtube.com/watch?v=nhvAAvwS-UA

[2] https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F

The biggest problem with the current popular idea of “free will” is that people think it means they’re ineffably unpredictable. They’re uncomfortable with the notion that if you were to simulate their brain in sufficient detail, you could predict thoughts and reaction. They take refuge in pseudoscientific mumbling about the links to the Quantum, for they have heard it is special and unpredictable.

And that’s just the polar opposite of having a meaningful will at all. It is good that you are pretty much deterministic. You shouldn’t be deciding meaningful things randomly. If you made 20 copies of yourself and asked them to support or oppose some essential and important political question (about human rights, or war, or what-have-you) they should all come down on the same side. What kind of a Will would that be that chose randomly?

Same. We are so, so profoundly not what it feels like we are, to most of us anyway.

I am morbidly curious how people are going to creatively explain away the more challenging insights AI gives us in to what consciousness is.

If that makes you uncomfortable you definitely should not go reading the evidence supporting the notion that conscious free will is an illusion.

https://www.mpg.de/research/unconscious-decisions-in-the-bra...

My impression is that the understanding of that research that comes up with statements like "But when it comes to decisions we tend to assume they are made by our conscious mind. This is questioned by our current findings" is based on dualistic reasoning.

The idea that there should not be any neural activity before a conscious decision is straight-up dualism---the intangible soul makes a decision and neural activity follows it to carry out the decision.

An alternative way of understanding that result is that the neural activity that precedes the "conscious decision" is the brain's mechanism of coming up with that decision. The "conscious mind" is the result of neural activity, right?

i always thought that the concept of free will didnt bother me. but it turns out i just didn’t understand what it implied. oh dear.

if it turns out that true, it’s truly amazing how well we convince ourselves that we’re in control.

but if our brain controls our actions and not our consciousness, then what is the purpose of consciousness?

Perhaps there is no purpose to consciousness.

Perhaps it's a phenomenon that somehow arises independently ex nihilo from sufficiently complex systems, only ever able to observe, unable to act.

Weird to think about.

Can conscious experience ever arise from matter? Even if the said matter is neural networks? This seems utterly nonsensical to me.

We are composed of matter and we are conscious, we take this as being axiomatic. Given we have one certain example, the question is then how does the body do it, and how could we do it in other substrates: how do we go from "just physics" to mind [1].

The short answer is: chemical reactions start a chain reaction of abstraction towards higher and higher forms of collective intelligence.

For some reason, perhaps something with the way the Hilbert space vector obeying the Schrödinger equation which we usually call "the universe" is [2], but also given the ridiculous breadth and depth of possible explorations of the biological pathways [3], "chunks" of matter tend to group together, they group and form stars and planets, but they also group and form formaldehydes and acetaldehydes and many more. Given enough tries, across multiple environments, in some lucky hydrothermal vents abiogenesis was probably started [4]. Once we had the first agential "chunk" of matter, a group of matter which has a definition of the boundary between a "self", no matter how tiny [5], and an exterior environment, it was more of a game of waiting (~4 billion years) for this tiny agent to grow into an agent with higher-order thinking, self-referentiality, metacognition, and the likes.

Neural networks, as in matrix multiplications, are not conscious because they have no mechanism for deciding what is the environment and what is their own, they are a hammer not a lacrymaria olor [6]. Could we have neural networks in an agent-like architecture starting to behave more like thermostats, setting goals for themselves? Probably.

[1] "From physics to mind - Prof. Michael Levin", https://youtu.be/_QICRPFWDpg?t=85

[2] "Sean Carroll: Extracting the universe from the wave function", https://www.youtube.com/watch?v=HOssfva2IBo

[3] "Nick Lane, 'Transformer : The Deep Chemistry of Life and Death'", https://youtu.be/bEFzUx_j7tA?t=279

[4] https://en.wikipedia.org/wiki/Hydrothermal_vent

[5] https://en.wikipedia.org/wiki/Physarum_polycephalum

[6] https://www.youtube.com/watch?v=sq6Y54mxjOg

Please see my reply to your sister comment. Thanks for the links, especially 1 btw.

Do you consist of matter? Are you conscious? Are you aware the brain is a neural network?

Let's assume the premise that a form of neural network is necessary but insufficient to give rise to conscious experience. Then might it not matter whether the medium is physical or digital?

If you answer this with anything other than "we don't yet know", then you'll be wrong, because you'll be asserting a position beyond what science is able to currently provide; all of this is an open question. But hint: the evidence is mounting that yes, the medium might not matter.

Once you take on an information theoretic view of consciousness, a lot of possibilities and avenues of research open up.

https://en.m.wikipedia.org/wiki/Neural_correlates_of_conscio...

Sorry, my english is not the best and I don't think there is a word for the thing I'm trying to explain. Meaning of 'consciousness' is too messy.

I know brain is a neural network. I just don't understand how cold, hard matter can result in this experience of consciousness we are living right now. The experience. Me. You. Perceiving. Right now.

I'm not talking about the relation between the brain and our conscious experience. It's obvious that brain is collecting and computing data every second for us to live this conscious experience. The very experience of perceiving, being conscious? The thing we take for granted the most, for that we're not without it any time, except when we're asleep?

Matter is what it is. A bunch of carbon and hydrogen atoms. How does the experience arise from matter? It can't. It is a bunch of atoms. I know how NNs and biological neurons work, still I don't see any way matter can do that. There must be some sort of non-matter essence, soul or something like that.

Is a bunch of electrochemical charges this thing/experience I am living right now? How can it be? Is Boltzmann brain [1] a sensible idea at all?

1: https://en.wikipedia.org/wiki/Boltzmann_brain

You are referring to qualia [1]: feeling something as-is, subjectively. I had similar metaphysics just a few years ago, but in the end, it's just that: bad metaphysics. And it's not even your or my fault: with 2,400+ years of bad metaphysics it's almost only luck to be able to pierce the veil into the metaphysics of tomorrow.

The main point is that with the tremendous discoveries of people such as Church/Turing (matter can be organized in such a fashion as to produce computation) [2] and those forgotten from the first page of history, such as Harold Saxton Burr (matter can be animated through bioelectricity), we no longer are bound to a static metaphysics where objects are made from a material which just sits there. It was obviously never the case, but fighting the phantasms of our own speculation is the hardest fight.

Therefore, no, matter is neither cold, nor hard, and we are surely very far from comprehending all the uses and forms of matter. Just look at all the wood objects around you and think how the same material was available also to Aristotle, 2,400 years ago, and to Descartes, 400 years ago, when they were writing their bad metaphysics, yet they were completely unable to think 1% of the wood objects you have readily available nowadays, cardboard and toothpicks included.

And also, yes, you are electrochemical charges, we all are, what else could we be? We looked insanely deep into the brain [4], there is no magic going on. A caveat would be that, yes, probably, we are not running on the first layer, at the level of sodium ions and neurotransmitters, but that the machinery, the brain, gives rise to a simulation: "only a simulation can be conscious, not physical systems" [5].

[1] https://en.wikipedia.org/wiki/Qualia

[2] https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis

[3] https://en.wikipedia.org/wiki/Harold_Saxton_Burr

[4] "The Insane Engineering of MRI Machines", https://www.youtube.com/watch?v=NlYXqRG7lus

[5] https://www.youtube.com/watch?v=tyVyirT6Cyk https://www.youtube.com/watch?v=SbNqHSjwhfs

I did not mean qualia, I knew qualia. I meant that my experience is not matter. Matter is matter and charges are charges, my experience right now is neither. Your perception, the act of you experiencing life right now is not matter.

> We looked insanely deep into the brain [4], there is no magic going on.

Indeed all computation and input collection and such happen in the brain. I just don't understand how I can experience anything if I'm composed only of matter. How come there happens to be a mind? Indeed the electrochemical charges from visual receptors in the eye will be transmitted and computed and memory and dopamine and all the neurons will fire regardless of whether I'm only matter or not. But how can the experiencing consciousness, 'me' arise from matter?

> only a simulation can be conscious, not physical systems

This is what I'm talking about, only that I don't see why simulations are not physical systems.

> yes, you are electrochemical charges, we all are, what else could we be?

It's nonsensical and unscientific to completely rule out the possibility that we can be something else as well, especially when we can't study it directly, like in the example of soul.

Simulations are physical systems just as much as running a "Hello World" program on your computer is a physical system: somewhere some transistors flip, but they are not relevant for the level of description we are interested when running the program, the program output, as simple or complex as it could be. Somewhere in the brain some molecules do "stuff", as a result of the "stuff" the brain sustains one agent, or more [1]. How exactly, in an engineering sense, the agent is constructed is yet to be discovered, hopefully we are only a few years, a few decades, away from building synthetic agents.

Sure, we have about 2,700 years of tradition speaking of souls (considering the major religions: Christianity, Islam, Buddhism, Hinduism, and Judaism). Where did those 2,700 years got us? Has any religion been able to build a conscious agent starting from basic materials (whatever they consider basic, pixie dust if they will)? Have all this years speaking of souls managed to achieve something meaningful, even as a side effect, that actually improves the quality of life? I'm talking hay [2], indoor plumbing, hook-and-loop fasteners, ibuprofen, GPS, voltmeters, extreme ultraviolet lithography, things that you and I can use and rely on daily. I have read pretty much all the major texts of the major traditions, from Mahābhārata to Summa Theologiae, call it intellectual curiosity. If not for the "bragging rights" to say that I know what filioque or bodhipakkhiyādhammā means, I would regret it, wasted time and pointless eye strain. So no, it's not nonsensical and unscientific to rule out a not even hypothesis such as the "soul" after 2,700+ years without any kind of results and absolute incompatibility with the way we actually interact with the world, scientifically or not: photons, atoms, electromagnetic fields and the like.

[1] https://en.wikipedia.org/wiki/Dissociative_identity_disorder

[2] "The technologies which have had the most profound effects on human life are usually simple. A good example of a simple technology with profound historical consequences is hay.", https://quotepark.com/quotes/1924489-freeman-dyson-like-many...

It’s apparent now that I’m unable to make you understand the concept I’m talking about. So I won’t try more.

To be clear on the other things you mentioned, I think science, enlightment and all the modern technology is completely worthless, evil, and harmful. I think they only degraded the quality of life, never increased. No, I’m not living under a rock, I know Rust and JS, have a digital ocean server and some domains, have built a cell phone from scratch, eat chocolate bars, have a 3 story concrete house, am surrounded by all sorts of digital screens. I’m aware what supposed “benefits” of modernity are.

See Islam [1,2] or the book Technological Slavery by Dr. David Skrbina [3].

[1] Documentary series: https://youtube.com/playlist?list=PLcnL9bB-q3ymnRNcpZGcVVjiD...

[2] Third-party moderated debates on Islam versus Modernity: https://youtube.com/playlist?list=PLcnL9bB-q3ylQ9CZdB8YfjoDv...

[3] https://ia800300.us.archive.org/21/items/tk-Technological-Sl...

Just wait until you notice how much humans do this day to day

Phantoms in the Brain is a fascinating book that deals with exactly this topic.

I'm not understanding the connection between your paragraphs here even after reading the first article.

Even if you accept classic theory (e.g. hemispheric localization and the homunculus) which most experts don't all this suggests is that the brain tries to make sense of the information it has and in sparse environments it fills in.

How does this make our behavior "mostly lies, fabrications, hallucinations, faulty re-memorization, post hoc reasoning" as most humans don't have a severed corpus callosum.

The discussion starts with:

"In a healthy human brain, these divergent hemispheric tendencies complement each other and create a balanced and flexible reasoning system. Working in unison, the left and right hemispheres can create inferences that have explanatory power and both internal and external consistency."

I think the point is that, in a non-healthy brain, the brain can create a balanced and flexible reasoning system that creates inferences that have explanatory power, but which may not match external reality. Oliver Sacks has a long bibliography of the weird things that can go on in brains.

But the bottom line is that introspection is not necessarily reliable.

He didn't say behavior. He said explanations of behaviour. Split brain experiments aside, this is pretty evident from other research. We can't recreate previous mental states, we just do a pretty good job (usually) of rationalizing decisions after the fact. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/

I'm reading this as our explanations for our own behavior as in why am I typing on this keyboard right now, in which case it's not evident at all.

The existence of cognitive dissonance suggested in your citation is in no way analogous to "our own explanations about ourselves and our behaviour are mostly lies, fabrications, hallucinations, faulty re-memorization, post hoc reasoning" and in fact supports the opposite.

One primary explanation of ourselves is that there is in fact a "self" there, we feel this "self" as being permanent, continuous through time, yet we are absolutely sure that is a lie: there are no continuous processes in the entire universe, energy itself is quantized.

In the morning when we wake up, we are "booting" up the memories the brain finds and we believe that we have persisted through time, from yesterday to today, yet we are absolutely sure that is a lie: just look at an Alzheimer patient.

We are feeling this self as if it's somewhere above the neck and we feel like this self is looking at the world and sees "out there", yet we are absolutely sure that is a lie: our senses are being overflown by inputs and the brain filters them, shapes a model of the world, and presents that model to the internal model of itself, which gets so immersed into model of the world that starts to believe the model is indeed the world, until the first bistable image [1] breaks the model down.

[1] https://www.researchgate.net/profile/Amanda-Parker-14/public...

Not supported by neuroimaging. Promoted without evidence or sufficient causal inference.

https://www.health.harvard.edu/blog/right-brainleft-brain-ri... :

> But, the evidence discounting the left/right brain concept is accumulating. According to a 2013 study from the University of Utah, brain scans demonstrate that activity is similar on both sides of the brain regardless of one's personality.

> They looked at the brain scans of more than 1,000 young people between the ages of 7 and 29 and divided different areas of the brain into 7,000 regions to determine whether one side of the brain was more active or connected than the other side. No evidence of "sidedness" was found. The authors concluded that the notion of some people being more left-brained or right-brained is more a figure of speech than an anatomically accurate description.

Here's wikipedia on the topic: "Lateralization of brain function" https://en.wikipedia.org/wiki/Lateralization_of_brain_functi...

Furthermore, "Neuropsychoanalysis" https://en.wikipedia.org/wiki/Neuropsychoanalysis

Neuropsychology: https://en.wikipedia.org/wiki/Neuropsychology

Personality psychology > ~Biophysiological: https://en.wikipedia.org/wiki/Personality_psychology

MBTI > Criticism: https://en.wikipedia.org/wiki/Myers%E2%80%93Briggs_Type_Indi...

Connectome: https://en.wikipedia.org/wiki/Connectome

This is not relevant to GP's comment. It has nothing to do with "are there fixed 'themes' that are operated in each hemisphere." It has to do with more generally, does the brain know what the brain is doing. The answer so far does not seem to be "yes."

Says who? There is actual evidence to support that our brain doesn't "know" what it is doing on a subconscious level? As far as I'm aware it's more that conscious humans don't understand how our brain works.

I think the correct statement is "so far the answer is we don't know"

The split brain experiments very very clearly indicate that different parts of the brain can independently conduct behavior and gain knowledge independently of other parts.

How or if this generalizes to healthy brains is not super clear, but it does actually provide a good explanatory model for all sorts of self-contradictory behavior (like addiction): the brain has many semi-independent “interests” that are jockeying for overall control of the organism’s behavior. These interests can be fully contradictory to each other.

Correct, ultimately we do not know. But it’s actually a different question than your rephrasing.

Given that functional localization varies widely from subject to subject per modern neuroimaging, how are split brain experiments more than crude attempts to confirm functional specialization (which is already confirmed without traumatically severing a corpus callosum) "hemispheric" or "lateral"?

Neuroimaging indicates high levels of redundancy and variance in spatiotemporal activation.

Studies of cortices and other tissues have already shown that much of the neural tissue of the brain is general purpose.

Why is executive functioning significantly but not exclusively in the tissue of the forebrain, the frontal lobes?

Because there’s a version of specialization that is, “different regions are specialized but they all seem to build consensus” and there’s a version that is “different regions are specialized and consensus does not seem to be necessary or potentially even usual or possible.”

These offer very different interpretations of cognition and behavior, and the split brain experiments point toward the latter.

Functional specialization > Major theories of the brain> Modularity or/and Distributive processing: https://en.wikipedia.org/wiki/Functional_specialization_(bra... :

> Modularity: [...] The difficulty with this theory is that in typical non-lesioned subjects, locations within the brain anatomy are similar but not completely identical. There is a strong defense for this inherent deficit in our ability to generalize when using functional localizing techniques (fMRI, PET etc.). To account for this problem, the coordinate-based Talairach and Tournoux stereotaxic system is widely used to compare subjects' results to a standard brain using an algorithm. Another solution using coordinates involves comparing brains using sulcal reference points. A slightly newer technique is to use functional landmarks, which combines sulcal and gyral landmarks (the groves and folds of the cortex) and then finding an area well known for its modularity such as the fusiform face area. This landmark area then serves to orient the researcher to the neighboring cortex. [7]

Is there a way to address the brain with space-filling curves around ~loci/landmarks? For brain2brain etc

FWIU, Markham's lab found that the brain is at max 11D in some places; But an electron wave model (in the time domain) may or must be sufficient according to psychoenergetics (Bearden)

> Distributive processing: [...] McIntosh's research suggests that human cognition involves interactions between the brain regions responsible for processes sensory information, such as vision, audition, and other mediating areas like the prefrontal cortex. McIntosh explains that modularity is mainly observed in sensory and motor systems, however, beyond these very receptors, modularity becomes "fuzzier" and you see the cross connections between systems increase.[33] He also illustrates that there is an overlapping of functional characteristics between the sensory and motor systems, where these regions are close to one another. These different neural interactions influence each other, where activity changes in one area influence other connected areas. With this, McIntosh suggest that if you only focus on activity in one area, you may miss the changes in other integrative areas.[33] Neural interactions can be measured using analysis of covariance in neuroimaging [...]

FWIU electrons are most appropriately modeled with Minkowski 4-space in the time-domain; (L^3)t

Neuroplasticity: https://en.wikipedia.org/wiki/Neuroplasticity :

> The adult brain is not entirely "hard-wired" with fixed neuronal circuits. There are many instances of cortical and subcortical rewiring of neuronal circuits in response to training as well as in response to injury.

> There is ample evidence [53] for the active, experience-dependent re-organization of the synaptic networks of the brain involving multiple inter-related structures including the cerebral cortex.[54] The specific details of how this process occurs at the molecular and ultrastructural levels are topics of active neuroscience research. The way experience can influence the synaptic organization of the brain is also the basis for a number of theories of brain function

"EM Wave Polarization Transductions" Lt. Col. T.E Bearden (1999) :

> Physical observation (via the transverse photon interaction) is the process given by applying the operator ∂/∂t to (L^3)t, yielding an L3 output.

"Representational drift: Emerging theories for continual learning and experimental future directions" (2022) https://www.sciencedirect.com/science/article/pii/S095943882... :

> Recent work has revealed that the neural activity patterns correlated with sensation, cognition, and action often are not stable and instead undergo large scale changes over days and weeks—a phenomenon called representational drift. Here, we highlight recent observations of drift, how drift is unlikely to be explained by experimental confounds, and how the brain can likely compensate for drift to allow stable computation. We propose that drift might have important roles in neural computation to allow continual learning, both for separating and relating memories that occur at distinct times. Finally, we present an outlook on future experimental directions that are needed to further characterize drift and to test emerging theories for drift's role in computation.

So, to run the same [fMRI, NIRS,] stimulus response activation observation/burn-in again weeks or months later with the same subjects is likely necessary given Representational drift.

You're ignoring several confounders and conflating several loosely related things that I'm not even sure what the point you're making is anymore.

To begin with, the split-brain experiments don't provide clear or strong evidence for anything given the small sample size, heterogeneity in procedure (i.e. was there complete comissurotomy or just callosotomy) and the elapsed time between neuropsychiatric evaluation and initial procedures which relies on the assumption that adaptation does not occur and neuroplasticity is not a thing. The split-brain experiments are notable because the lab experiments SUGGEST the lack of communication between two hemispheres and a split conscious however this is paradoxical with everyday experience of these patients, far from providing evidence for anything.

Ignoring that for a moment, how do the split-brain experiments support 'the brain has many semi-independent “interests” that are jockeying for overall control of the organism’s behavior'?

How is addiction self-contradictory exactly and what does this have to do with split-brain?

If your point is that different parts of the brain (e.g. the reward system and the executive function regions) have different roles this isn't really debatable, obviously different parts of the brain are all doing their individual jobs and the most widely accepted theory is that these are integrated in some unknown mechanism by a single consciousness which remains in control.

Your original statement of: does the brain know what the brain is doing. The answer so far does not seem to be "yes."

Suggests you're arguing that the brain has many different consciouses that are in a constant battle, i.e. there is not a unified consciousness in control of behavior.

To take your addiction example, the brain is very much aware of what it is doing and addiction is not self-contradictory because short term rewards are being valued above long-term ones and health of the organism. The reward system model provides an excellent neurobiological explanation for addiction.

This is not directly evidenced by either addiction or the split brain experiments and is at best a hypothetical model hence why it's a theory and my original response to your statement.

I'm not "ignoring" them at all. I'm saying that they point to interesting questions that are not answered.

> The split-brain experiments are notable because the lab experiments SUGGEST the lack of communication between two hemispheres and a split conscious however this is paradoxical with everyday experience of these patients, far from providing evidence for anything.

It is not "paradoxical" but yes it does conflict with some reported experience. However, even healthy individuals often report being "of two minds" or struggling to "make up their [singular] mind." Why are these utterances to be dismissed while the also-subjectively-reported sensation of unitary experience is taken as fact?

> Suggests you're arguing that the brain has many different consciouses that are in a constant battle, i.e. there is not a unified consciousness in control of behavior.

I wouldn't characterize my position as "many different consciousnesses," but rather that consciousness is dispersed across (at least) the brain. In some scenarios (such as a corpus callosotomy) and perhaps in more everyday scenarios - perhaps all day every day - that dispersed activity can fall out of internal "synchronization." Anyway, you provided the exact same interpretation in the previously quoted section: "the lab experiments SUGGEST the lack of communication and a split consciousness."

You just go one step further of prioritizing the subjectively reported sensation of unitary consciousness over also-subjectively-reported sensation of non-unitary consciousness. That's your prior taking hold, not mine, and not actual evidence.

You yourself admit we do not know the mechanism (if any exists) by which the activity in various parts of the brain are integrated. We do not know if this process actually even occurs!

Regarding addiction, it is very, very commonly reported that addicts will go into "autopilot" like states while satisfying their addictions and only "emerge" when they have to face consequences of their behaviors. Again, subjectively reported, but so is the experience of unitary consciousness! If we cannot trust one then we shouldn't take it as granted that we can trust the other.

I get the sense you think you're arguing against some firmly held belief or a model I'm proposing as fact: you're not! We're both saying "we don't know much about how this works." And no, neurochemical mechanisms are not complete answers to how brain activity ladders up to conscious experience, similar to how a molecular model of combustion cannot explain much about urban traffic patterns.

I'm arguing against the strength of your statements based on methodologically unsound experiments that do not "very very clearly indicate" anything beyond pose a few questions for which there are several different hypothetical answers. All of which have zero evidence behind them.

Similarly, the initial comment of 'does the brain know what the brain is doing. The answer so far does not seem to be "yes."' is misleadingly suggesting there is a shred of evidence supporting that the answer is 'no' or that the answer is 'not yes'. There are no answers so far, just questions.

If anything, there are more unified consciousness hypotheses than otherwise, although if you refer back to my original reply I did not make this assertion: 'I think the correct statement is "so far the answer is we don't know"'

> It is not "paradoxical" but yes it does conflict with some reported experience.

Rather than belabour the experiment results and implications here is a great peer-reviewed article by experts in the field: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/

> Regarding addiction, it is very, very commonly reported that addicts will go into "autopilot" like states while satisfying their addictions and only "emerge" when they have to face consequences of their behaviors. Again, subjectively reported, but so is the experience of unitary consciousness! If we cannot trust one then we shouldn't take it as granted that we can trust the other.

The dopamine reward system understanding (which by the way is probably the most well-validated and widely believed model in neurobiology) provides a rational explanation for addiction.

You haven't explained what is self-contradictory, that a few case reports exist of patients claiming they went in and out of consciousness? That's not a contradiction.

The split brain experiments clearly indicate exactly what I said they clearly indicate. To refresh: "that different parts of the brain can independently conduct behavior and gain knowledge independently of other parts".

From your own linked article: "In short, callosotomy leads to a broad breakdown of functional integration ranging from perception to attention."

If the dopamine reward system is the full answer, then what explains drug addicts anguishing about their addiction while simultaneously actively seeking out their next hit? What part of the brain is producing the anguish if the whole behavior is conclusively described by the dopamine reward system?

Breakdown of functional integration =/= different parts of the brain can independently conduct behavior and gain knowledge.

Still not sure what the contradiction is, is it regret now? Cause that isn’t contradictory.

It’s pretty simple, drugs feel really good when you take them so a single consciousness prioritizes that feeling over long term interests. When one is not taking them and facing the consequences of those decisions they feel bad. To make the bad feelings go away one takes more drugs and the cycle repeats.

You are talking about the popular narrative of “left brain” thinking being more logical and “right brain” thinking being more creative. You are correct this is unsupported.

The post you are replying to is talking about the small subset of individuals who have had their corpus callosum surgically severed, which makes it much more difficult for the brain to send messages between hemispheres. These patients exhibit “split brain” behavior that is well studied by experiments and can shed light into human consciousness and rationality.

Your response doesn't seem to be directly related to the previous poster's split-brain comments, but rather the popular misuse of the lateralization idea.

Agree I'm not a neuro subspecialist but I've listened to some talks at conferences out of interest and I don't think anyone still believes in this anymore. Anecdotally the few fMRI's I reported as a trainee didn't support this either.

"LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own."

There is no "their" and there is no "thought process" . There is something that produces text that appears to humans like there is something like thought going on (cf the Eliza Effect), but we must be wary of this anthropomorphising language.

There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text.

As we don't know for sure what is happening 100% within a neural network, we can say we don't believe that they're thinking and we would still need to define the word thinking. Once LLM's can self-modify, the word "thinking" will be more accurate than it is today.

And when Hinton says at MIT, "I find it very hard to believe that they don't have semantics when they consult problems like you know how I paint the rooms how I get all the rooms in my house to be painted white in two years time," I believe he's commenting on the ability of LLM's to think on some level.

In this case, I think we do if you will check out the paper (https://openaipublic.blob.core.windows.net/neuron-explainer/...). Their method is to

1. Show GPT-4 a GPT-produced text with the activation level of a specific neuron at the time it was producing that part of the text highlighted. They then ask GPT-4 for an explanation of what the neuron is doing.

Text: "...mathematics is done _properly_, it...if it's done _right_. (Take ..."

GPT produces "words and phrases related to performing actions correctly or properly".

2. Based on the explanation, get GPT to guess how strong the neuron activates on a new text.

"Assuming that the neuron activates on words and phrases related to performing actions correctly or properly. GPT-4 guesses how strongly the neuron responds at each token: '...Boot. When done _correctly_, "Secure...'"

3. Compare those predictions to the actual activations of the neuron on the text to generate a score.

So there is no introspection going on.

They say, "We applied our method to all MLP neurons in GPT-2 XL [out of 1.5B?]. We found over 1,000 neurons with explanations that scored at least 0.8, meaning that according to GPT-4 they account for most of the neuron's top-activating behavior." But they also mention, "However, we found that both GPT-4-based and human contractor explanations still score poorly in absolute terms. When looking at neurons, we also found the typical neuron appeared quite polysemantic."

Or maybe the human thought process isn't as sophisticated as we imagined.

I'm not arguing for or against that. It's more the implications of sentience and selfhood implicit in the language many use around LLMs.

What if you ask it to emit the reflexive output, then feed that reflexive output back into the LLM for the conscious answer?

What if you ask it to synthesize multiple internal streams of thought, for an ensemble of interior monologues, then have all those argue with each other using logic and then present a high level answer from that panoply of answers?

What if you do? LLMs don't have reflexive output or internal streams of thought, they are simply (complex) processes that produce streams of tokens based on an inputted stream of tokens. They don't have a special response to tokens that indicate higher-level thinking to humans.

If you direct the model output to itself and don't view it otherwise, how is it not an "internal stream of thought"?

LLMs seem to me to be the "internal streams of thought". I.e. it's not LLMs that are missing an internal process that humans have, but rather it's humans that have an entire process of conscious thinking built on top of something akin to LLM.

I agree completely and I think this is where a lot of people get tripped up. There's no reason to think an AGI needs to be an LLM alone, it might just be a key building block.

Well put, and I agree. My belief is that if a typical person was drugged or otherwise induced to just blurt out their unfiltered thoughts out loud as it crossed their brain, the level of incohesion and false confidence on display would look a lot like an LLM hallucinating.

That's possible I guess but is there positive evidence for that being the case?

The way I phrased it isn't exactly structured to admit any kind of evidence, so let me destructure it. My observation is that:

- The argument that LLMs are missing introspection / inner voice is based on attempting to compare LLMs directly with human minds.

- Human minds have conscious and unconscious parts; for many people, part of the boundary between unconscious and conscious mind manifests as the "inner voice" - the one that makes verbalized thoughts "appear" in their head (or rather perhaps become consciously observed).

- Based entirely on my own experience in using GPT-3.5 and GPT-4, and my own introspection, I feel that GPT-4 bears a lot of resemblance to my inner voice in terms of functioning.

- Therefore I propose that comparing LLMs directly to human minds is unproductive, and it's much more interesting/useful to compare them to the inner voice in human minds: the part of the boundary between unconscious and conscious that uses natural language for I/O.

You seem to have high confidence in how LLMs work.

The text output of a llm is the thought process. In this context the main difference between humans and llms, is that llms can’t have internalized thoughts. There are of course other differences to, like the fact that humans have a wider gamut of input: visuals, sound, input from other bodily functions. And the fact that we have live training.

It's not clear whether or not LLMs have internal thoughts -- each token generation could absolutely have a ton of thought-like modelling in the hidden layers of the network.

What is known is that these internal thoughts get erased each time a new token is generated. That is, it's starting from scratch from the contents of the text each time it generates a word. But you could postulate that similar prompt text leads to similar "thoughts" and/or navigation of the concept web, and therefore the thoughts are continuous in a sense.

True, LLMs definetly has something that is "thought-like".

But todays networks lacks the recursion(feedback where the output can go directly to the input) that is needed for the type of internalized thoughts that humans have. I guess this is one thing you are pointing at by mentioning the continuousnes of the internals of LLMs.

Pride comes before the fall, and the AI comes before humility.

Precisely.

When someone states definitively what LLMs can or cannot do, that is when you know to immediately disregard them as the waffling of uninformed laymen lacking the necessary knowledge foundations (cognitive/neuroscience/philosophy) to even appreciate the uncertainty and finer points under discussion (all the open questions regarding human cognition etc).

They don't know what they don't know and make unfounded assertions as result.

Many would do to refrain from speaking so surely about matters they know nothing about, but that is the internet for you.

https://en.wikipedia.org/wiki/Predictive_coding

> There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text.

To be clear, you're saying that we should just dismiss out-of-hand any possibility that an LM AI might actually be able to explain its reasoning step-by-step?

I find it kind of charming actually how so many humans are just so darn sure that they have their own special kind of cognition that could never be replicated. Not even with 175,000,000,000 calculations for every word generated.

That's a strawman since I didn't argue anything about humans being special. I don't think there is anything necessarily inherently special about human intelligence, I'm just advocating for caution around the language we use to talk about current systems.

All this talk of AGI and sentience and so on is premature and totally unfounded . It's pure sci fi, for now at least.

> I didn't argue anything about humans being special

Above you said about AI LMs:

> There is no "their" and there is no "thought process"

So, unless you're claiming that humans lack a thought process as well, then you're arguing that humans are special.

> All this talk of AGI and sentience and so on is premature and totally unfounded

I don't see any mention of AGI or sentience in this thread?

Also, I don't think anyone could read this transcript with GPT-4 and still claim that it's incapable of a significant degree of self-reflection and metacognition:

https://news.ycombinator.com/item?id=35880148

I reject the words 'self' and 'cognition' in your comment. This is exactly what I'm talking about. A facsimile of them, maybe...

Very true. In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal.

I think LLMs are "Semantic Clouds of Words" + grammar and syntax generator. Someone could just discard the grammar and syntax generator, just use the semantic cloud and create the grammar and syntax by himself.

For example, in writing a legal document, a slightly educated person on the subject, could just use the relevant words put into an empty paper, fill in the blanks of syntax and grammar, alongside with the human reasoning which is far superior than any machine reasoning, till today at least.

The process of editing the GPT* generated documents to fix reasoning is not a negligible task anyway. Sam Altman mentioned that: "the machine has some kind of reasoning", not a human reasoning ability by any means.

My point is, that LLMs are two programs fused into one, "word clouds" and "syntax and grammar", sprinkled with some kind of poor reasoning. Their word clouding ability, is so unbelievable stronger than any human it fills me with awe every time i use it. Everything else is, just whatever!

I think they're much more than that. Or rather, if they're "Semantic Cloud of Words", they're still a hundred thousand dimensional clouds of words, and in those hundred thousand dimensions, any relationship you can think of, no matter how obscure, ends up being reflected as proximity along some subset of dimensions.

Looking at it this way, I honestly wouldn't be surprised if that's exactly how "System 1" (to borrow a term from Kahneman) in our brains works.

What I'm saying is:

> In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal.

It may be much more of a deal than we'd naively think - it seems to me that a lot of what we'd consider "thinking" and "reasoning" can be effectively implemented as proximity search in a high-dimensional enough vector space. In that case, such extracted "Semantic Cloud of Words" may turn out to represent the very structure of reasoning as humans do it - structure implicitly encoded in all the text that was used as training data for the LLMs.

>if they're "Semantic Cloud of Words", they're still a hundred thousand dimensional clouds of words, and in those hundred thousand dimensions, any relationship you can think of, no matter how obscure, ends up being reflected as proximity along some subset of dimensions.

Yes, exactly that. That's what GPT4 is doing, over billions of parameters, and many layers stacked on top of one another.

Let me give you one more tangible example. Suppose Stable Diffusion had two steps of generating images with humans in it. One step, is taking as input an SVG file, with some simple lines which describe the human anatomy, with body position, joints, dots as eyes etc. Something very simple xkcd style. From then on, it generates the full human which corresponds to exactly the input SVG.

Instead of SD being a single model, it could be multimodal, and it should work a lot better in that respect. Every image generator suffers from that problem, human anatomy is very difficult to get right.[1] The same way GPT4 could function as well. Being multimodal instead of a single model, with the two steps discreet from one another.

So, in some use cases, we could generate some semantic clouds, and generate syntax and grammar as a second step. And if we don't care that much about perfect syntax and grammar, we feed it to GPT2, which is much cheaper to run, and much faster. When i used the paid service of GPT3, back in 2020, the Ada model, was the worst one, but it was the cheapest and fastest. And it was fast. I mean instantaneous.

>the very structure of reasoning as humans do it

I don't agree that the machine reasons even close to a human as of today. It will get better of course over time. However in some not so frequent cases, it comes close. Some times, it seems like it, but only superficially i would argue. Upon closer inspection the machine spits out non sense.

[1] Human anatomy, is very difficult to get right, like an artist. Many/all of the artists, point out the fact, that A.I. art doesn't have soul in the pictures. I share the same sentiment.

Clearly there is some process going on to decide what word to pick. Why can’t we call that thinking?

This is it. This comprehension of the chats is symptom of something like linguistic pareidolia. It's an enforced face that is composed of some probabilistic incidents and wistfulness.

And if the LLM is the explainer, it can lie to us if 'needed'.

I’m sure LLMs are quickly going to learn to hallucinate (or let’s use the proper word for what they’re doing: confabulate) plausible-sounding but nonsense explanations of their thought processes at least as well as humans.

I don't think we invent new words just for AI to explain its thought process to us better. AI may explain more elaborately in our language instead.