You might be redefining words here; there isn't a form of intelligence that isn't actual intelligence. It is all actual intelligence. Artificial in this context means it is something we're creating in a lab. LLMs can't avoid being artificial intelligence. The meaning of "AI" is to artificially create actual intelligence.
And if anything, average AI user is vastly overstating how good/useful it is. Papers about it pretty much always show huge gap between "productivity person thinks they are achieving" and "actual growth of productivity"
A lot of human intelligence is really societal rather than individual, based on knowledge transmitted down through generations by writing (the real enabler). If you take that away then what you are left with is something more like an isolated hunter-gather tribe.
Your point about writing and social intelligence is, to me, more evidence for the "it's language that's smart, not us" hypothesis. We start off in small bands of hunter-gatherers that store their intelligence in an oral culture. Language then jumps to clay tablets, papyrus, codex books, etc. The printing press allows it to escape containment to a wider public than just a caste of priests and bureaucrats. As soon as we invent automatic calculators, we start networking them and using those to process language, albeit in a primitive way (email, the web, etc.). Recently we discovered some abstruse math that, with the assistance of a bunch of beefy video cards, can crunch centuries of human writing into a mathematical object that encodes at least some of the meaning of that writing into an even more "advanced" symbolic processing machine. There's a clear trajectory of language itself getting more and more free of the specific wetware it grew up on.
It's a falsifiable claim, in that if there is a way to train a useful LLM from scratch without any human authored input language to bootstrap it (something I've been on the lookout for but haven't seen, though admittedly I'm not an AI researcher, just some Linux nerd with a day job as an SRE), then we can disprove it.
For the religious angle, look no further than John 1:
"In the beginning was the Word, and the Word was with God, and the Word was God."
(This is admittedly less falsifiable!)
I'm sure that we will eventually build artificial brains, capable of bootstrapping communications and language for themelves (if run en-masse in a simulation where the benefit of communication would emerge). An LLM can't do this since it is by definition/construction something only capable of learning a pre-existing language.
An artificial brain, just like a wet jiggly one, is always going to be more intelligent than a one-trick pony like an LLM - a language processor, but it is notable how intelligent that one-trick pony nonetheless appears to be.
You could choose to lock it in a virtual or physical basement with printout of the Common Crawl dataset and raise it like an LLM that learns language with zero real world grounding, in which case it may feel a bit like an LLM (but smarter - able to learn, etc), or you could let it interact with the real world and learn everything, including language, that way and be a lot more grounded.
It's hard to guess how the grounded version would feel to talk to vs the CommonCrawl one - I think it would mostly come down to how far we wanted to go in making the artificial brain have all the moving parts of a human one. In an odd way the more human we tried to make it, the more alien it might feel, but not for the "uncanny valley" reason you might imagine...
The thing with an LLM that makes it feel so human is that they are designed to 100% copy humans - their output is 100% driven by the training goal of trying to exactly match the samples in the training set. As soon as we start to try to build something more brain-like then it's behavior is going to be a lot less predictable - not just auto-regressive "auto-pilot speech", but driven by it's own internal thoughts, emotions, innate traits, etc - depending on how much of our brain we tried to copy in the artificial one. I expect it would feel a lot more organic, less robotic, to talk to, but at the same time perhaps less human since unlike the LLM it's not built just to mimic human speech.
If LLMs had shame, they'd surely not repeat mistakes (in the same context window) as much as they do.
People love to put a lot of meaning on what an LLM responds with when asked why it made a mistake, but it's critical to remember that the answer to that prompt is just another series of probabilistic tokens, and has no actual relation to how the error happened.
The LLM does not understand itself in any way.
That's reasonable, but it doesn't mean that LLMs are close to being brains.
For a start, when humans think/talk, we often think ABOUT something - whatever is swirling about in our mind, or what we are currently seeing/feeling/etc. An LLM generating tokens/words is doing so only based on it's weights and the word sequence it is currently generating ... the human parallel would be more like a rapper spitting out words based on prior words, essentially on auto-pilot, or when we get triggered into spitting out stock phrases like "have a nice day".
If you want to compare an LLM to a human brain, it's basically equivalent to our language cortex if you ripped out all the external connections and ripped out all the feedback paths that make it capable of learning.
Of course there is a lot more to our brain than just our language cortex, but that alone should make you realize there is no real comparison beyond the fact that our language generation is also going to be based on prediction, and partly auto-regressive.
Think about how much your own writing (and programming, if you were lucky enough to start early) evolved from, say, age 12 (when a lot of smart kids start to tackle 'real' books) to age 18 (when you supposedly have a good enough education for 50% of work in most countries) to age 25.
All of that evolution is a direct result of one thing: practice! But with a magic answer box available in everyone's pocket, it'll take truly Herculean effort from a learner to actually grind through the practice instead of just cheating for an answer. I really worry how much an LLM user will actually comprehend their own code or even prose; if you've scarcely written a line of code, how can you really understand what's going on in a debugger? If you haven't done the legwork of writing essays and constructing coherent arguments and comprehending grammar, how will you ever communicate effectively?
Maybe I'm just a dinosaur and these kids will sail a whole level of abstraction above my own understanding of writing and programming, much like how my own generation preferred Python to C, and how the previous generation evolved from assembly to C/BASIC/etc. But then I come back to those missing fundamentals, that empty mental model. It's not like my English or CS teachers had me grind through essays and implementing linked lists and Djikstra's Algorithm for pure busywork. They did it because practice is the only way to truly immerse a student in a practical subject. Maybe it'll work for programming, as long as LLMs get good enough that you can always ask them to fix low-level errors for you? But it seems unlikely to work in prose. And even those generational programming jumps I mentioned (assembly to C to Python) were lossy; most kids I went to school with would be absolutely useless writing C code, and even as a bit of a dinosaur I'm pretty awful at even debugging assembly.
Like you said: you still need to learn grammar and spelling. And I suspect a whole skill tree of other fundamentals!
It still gets things wrong, I can tell as I get through problems.
But it was either that or that dreary 'Cracking the Coding Interview' book. At least I'm learning fundamentals by asking question after question and making it track the concepts I had trouble with.
That's one use. Will most people use it to learn? Probably not. But most people are ... most people.
AI will be there, but it'll transform. When I say I don't use AI (i.e. LLMs, chat interfaces, agents and "autocomplete") for coding, research and whatnot, people label me as a luddite. The fact is I know how to use them. I test them from time to time. Occasionally these tools help. More often they hinder.
"Resistance is futile, hand your brain over!" is a hype filled dystopian fatalism noting that future is inevitable. It's inevitable. You can use this correctly, and we don't got back to our senses to understand how to use this correctly and efficiently.
We are just cooking our planet right now, with heat, poisoned water and slop.