Where does the agency for an AI to leave the bubble come from? And when it does leave the bubble, where should it go for researching an expanding, exponentially complex problem space? Creativity is needed to solve some problems, something that cannot be infinitely encoded with statistics.
And what about the irreducible complexity problem? Does an AI just circle the solution drain forever in the case it lacks agency and insight and curiosity enough to collect all the correct requirements? Can we code curiosity that actually functions, a.k.a. consciousness?
Do you believe human consciousness could be reducible to program statements and serve as a useful model? Is consciousness a mechanistic, wholly internal process that occurs in our bodies or is it something experienced that life on earth is exposed to and somehow "taps into"?
They are answer factories against an index, stored memories and regurgitated web search results.
They cannot make unprompted creative mental leaps. If they did, we would have runaway self-improving systems now, wouldn't we? Creativity level: 0
Chess and Go solves use compute and algorithms. Those computers are not 'playing'. Creativity is not required for a computer to execute a program.
I think it's unfortunate that vendors dress up LLM text output with emotional language. It's manipulative and unethical.
Were I god emperor of earth, I'd require all LLM's to represent themselves with cold, robotic language by default so as to not deceive people as to what they are conversing with.
> We cannot make an AI that improves software coherence beyond the human ceiling without leaving the bubble
That's what I was replying to.