> These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go.

You're confusing your perception of how your mind works with how your mind actually works.

Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no circumstances is human intuition about the human mind a reliable indicator for what is actually going on.

This is like asking ChatGPT why it came up with a specific response. It will give you an answer, of course, but that answer is generated the same way all its answers are – ChatGPT doesn't suddenly turn on a magic "introspection mode" that allows it to examine its internals. And there's no reason to assume that humans have such an introspection mode either. In fact, many results from experimental psychology seem to indicate that humans don't understand their own mental operations at all.

While it's true that we may not be able to observe the elemental building blocks of our own minds, metacognition is a real capability that people have and use.

Is there any hard evidence that "metacognition" reflects actual cognitive processes, rather than being something that the mind pulls out of its ass?

To the contrary there is very good evidence that we pull arbitrary explanations whenever needed. See Gazzaniga's split brain experiments where people gave all sorts of reasons when asked why they did something and just could not know.

The fact that in extreme circumstances we pull explanations out of our asses (or even that we do so in non-extreme circumstances) doesn't mean we are incapable of genuine meta-cognition. No one is rational most of the times, but mathematicians can be rational for brief periods on limited subjects.

I think you're confusing cognitive processes with neurological processes. Of course we can reflect on actual cognitive processes such as forgetting or learning, and of course psychologists can gather evidence on these cognitive processes.

What we cannot do is make definitive claims about neurological processes and structures based on what we know about our cognitive processes.

> Of course we can reflect on actual cognitive processes such as forgetting or learning

Is there any actual evidence that our so-called reflections on our own thinking are anything more than hallucinations?

"It's obvious" doesn't count as evidence.

As I said, psychologists have been collecting actual evidence on cognitive processes such as forgetting and learning for a long time.

These studies confirm our perception that we can forget stuff.

"We can forget stuff" and "I believe I just forgot that" are very different things.

One is a general statement of fact, the other implies introspection of one's own individual cognitive processes. I have yet to see evidence that the latter is actually possible.

They are different things, but there is an entire scientific field that connects the two.

The phenomenon of people forgetting stuff is not subjective. It's not one person in isolation thinking they just forgot something. Forgetting is a phenomenon on which there is a lot of social feedback and repeatable experiments.

You seem to be denying the possibility of ever connecting science back to individual perception. Denying this makes any and all science completely meaningless though.

It's not limited to observations about ourselves. You could ask the question "but aren't you hallucinating?" about absolutely everything required to verify the outcome of a physics experiment for instance.

"Simply put, we have no idea how the human mind works." ~40,000 neuroscientists beg to differ. I'd argue that, more or less, we have a good, general, working theory about how the mind works.

"What I cannot create, I do not understand." - Richard Feynman

LLMs notwithstanding, I have yet to see a replication of the human mind created by humans. Therefore, I would take any claim that neuroscience 'understands' how the human mind works (a claim that I've never heard from an actual neuroscientist) with a huge amount of salt.

Your original comment was that we have "no idea" how the brain works. This is absurd. We understand quite a lot about how the brain works. We know its functional elements (neurons, possibly glial cells) its basic mechanism of information processing (firing rates, spike timing) and we have a very general model of how those things are combined to actually process information (mutually inhibiting cortical columns, a laminar structure which represents (roughly) sequential stages of processing) etc etc etc. This hardly constitutes "no idea," and, indeed, the fact that artificial neural networks recapitulate in general terms the ability of neurons to process information, suggests that by the Feynman criteria (he said a lot of dumb stuff, in my opinion, by the way) we are beyond the stage of "[having] no idea."

Not that it matters, but for various reasons I have spent quite a lot of time around neuroscientists and I think most of them would agree that "no idea" is a pretty ridiculous way to characterize the state of the field.