But if a human is allowed time and internal reasoning iterations, so should the LLM when determining if it has deep insight. Right now we're simply observing input -> output of LLMs, the equivalent of snap answers from a human. But nothing says it couldn't instead be an input -> extensive internal dialogue, maybe even between multiple expert models for seconds, minutes or hours, that are not at all visible to the prompter -> final insightful answer. Maybe future LLMs will say, "let me get back to you on that".
From a computer science point of view: a single prompt/response cycle from a LLM is equivalent to a pure function; the answer is a function of the prompt and the model weights and is fundamentally reducible to solving a big math equation (in which each model parameter is a term.)
It seems almost self evident that "reasoning" worthy of the name would involve some sort of iterative/recursive search process, invoking the model and storing/reflecting/improving on answers methodically.
There's been a lot of movement in this direction with tree-of-thought/chain-of-thought/graph-of-thought prompting, and I would bet that if/when we get AGI, it's a result of getting the right recursive prompting pattern + retrieval patterns + ensemble models figured out, not just making ever-more-powerful transformer models (thought that would certainly play a role too.)
The LLM isn't the whole brain. Just the area responsible for language and cultural memory.
“You learn best from your mistakes” comes to mind here.
And not just shallow feedback loops but bigger, more profound ones eg “I didn’t go to work for a month and so I was fired and and now I have no money and it will now be hard to be hired again”.
Or we could just give them three laws
Otherwise the laws will have to be implemented as weights or during training so the model explicitly knows the laws and would never even be capable of doing something against them.
I suppose the real wisdom there is that humans are doomed to fail at alignment if we create sentience and expect it only to serve us.
If the shim was just another AI, then how do you align that AI? Who watches the watchers? But if it was a deterministic algorithm it would probably fail for the same reasons that algorithmic AI never went anywhere.
spoilers warning:
Is that basically the plot to westworld?
The great part is with clear enough directions it also knows how to evaluate whether its done or not.
Or am I misunderstanding something about that technique?
I guess from that perspective it might make sense to test if such a thing is already happening in current LLM's and my dismissive attitude stems from the fact that I've played with them enough to know that they currently don't.
We can also potentially add longer term memory - summarise the context, and judge which parts are important and stuff them in a vector store, and now and again swap in similar pieces of past context.
But of course it's not either or - better prompting to get the LLMs to do better from the start doesn't compete with then feeding that into an external loop as well.
No, it's the equivalent of putting a gun to someone's head and asking them "what are my intentions?" Which is readily available to any being with a theory of mind.
Because, obviously, training data probably includes a decent amount of motivation breakdowns as a function of coercion.
It doesn’t know why, but it knows what to say.
Are they faking it? Will they murder me? Are they just trying to scare me?
I don't understand the point you're trying to make.
Put gun to persons ahead.
Ask them to do a division.
Then screaming at them "HOW DID YOU DO THAT, TELL ME NOW, OR YOU'RE TOAST".
Even most humans would splutter and not be able to answer.
A human also can't explain how their neurons calculated a division, or anything else, or 'intention', even if a gun is pointed at them.
So, why is that a fault of AI if it can't explain how itself is working? It is not a requirement for AGI.
you :"Which is readily available to any being with a theory of mind."
I'm saying humans do not have this ability. We don't have a valid proven theory of mind, and even if we did, that knowledge would not change how our brains process inputs, and somehow give us access to how we processed an answer. We don't know our 'intention'
These responses are logically the same as "No You".
Of course, yes, we do know one thing that's missing in LLMs, which is "loop and helpers" like you describe. Which I'm sure many people are currently hacking at - one way being for the LLM to talk to itself.
But as for "a theory of mind", if enough writings served as input, then LLMs do have plenty of that.
Another question is whether LLMs are raised to behave like humans (which might be where they most NEED some theory of mind). Of course not. The ones we know most about are only question answerers. The theory of mind they might have (that is not negated by the lack of loop and internal deliberation) may be overwhelmed by the pre- and post-processing: "no sex, no murder plots, talk to the human like they are 5, bla bla bla". And yet you can ask things like "Tell it like you are speaking to 5 year olds who want to have a fun time". Some theory of mind makes it through.
While this is a simple example, the concept extrapolates to more complex issues as we develop cognitively and emotionally. Not only is theory of mind about recognizing that your thoughts and feelings are separate and distinct, but also being able to project your understanding into others to predict how they might be thinking and feeling in their own separate circumstance. This is the fundamental basis of empathy, or being able to predict and understand how others around you may be feeling given their unique current circumstances even if they are different from your own.
LLMs have not demonstrated any of the above understanding. Again, I'm sure they could generate a definition for it, and on any given prompt they could probably even generate some generic-ish text that could pose as empathy, but so can a horoscope. But there is not the deep, continuous comprehension of the user as a separate thinking and feeling agent that is fundamental to the idea of theory of mind.
After that, the entire field of AI is about making bits and register transfers - a written form of knowledge - into something indistinguishable from the wet version. Making a priori and irreconcilably different "academic understanding" and "functional, meaningful understanding" is going to be a problem.
I bet a lot of these experiments would already solvable by putting the LLM in a simple loop with some helper prompts that make it restructure and validate its answers, form theories and get to explore multiple lines of thought.
If an LLM would be able to do that in a single prompt, without a loop (so the LLM always answers in a predictable amount of time), then it would mean its entire reasoning structure is repeated horizontally through the layers of its architecture. That would be both limiting (i.e. limit the depth of the reasoning to the width of the network) and very expensive to train.