I had a similar insight (blog post: https://kristiandupont.medium.com/empathy-articulated-750a66...).

In a very unscientific way, I would say that the LLM is not the whole brain, it's part of it and we are still in the process of simulating other parts. But it does seem to me like we've solved the hard part, and it's astonishing to me that people like authors of this article seem to think that the current state of things is where evolution stops.

The brain works through flow of stimulation. Inputs and outputs probably creating consciousness somehow. LLMs are limited in self stimulation and it's memory model is very basic compare to humans.

But I feel most people come with gut feeling position that we have divine spark that makes us tick. And the view that we might be more basic even crude akin to LLMs makes them uneasy.

No it is not neither eerily nor computationaly similar to AI hallucinations, as we do not have a dam idea how the brains think.

"You are Two" (CGP Grey)

https://www.youtube.com/watch?v=wfYbgdo8e-8