News for hackers. Stuff that matters.
Re:Large language models lack deep insights or a theory of mind by tinco on Thursday November 30, 2023 @12:39PM (#38476403)
I think that if they would, that would be very surprising and indicative of a lot of wastefulness inside the model architecture. All these tests are simple single prompt experiments, so the LLM's get no chance to reason about their responses. They're just system 1 thinking, the equivalent of putting a gun to someone's head and asking them to solve a large division in 2 seconds.

I bet a lot of these experiments would already solvable by putting the LLM in a simple loop with some helper prompts that make it restructure and validate its answers, form theories and get to explore multiple lines of thought.

If an LLM would be able to do that in a single prompt, without a loop (so the LLM always answers in a predictable amount of time), then it would mean its entire reasoning structure is repeated horizontally through the layers of its architecture. That would be both limiting (i.e. limit the depth of the reasoning to the width of the network) and very expensive to train.

Re:Large language models lack deep insights or a theory of mind by Androider on Thursday November 30, 2023 @01:01PM (#38476685)
The equivalent for a human would be an reflexive response to a question, the kind you could immediately answer after being woken up at 3am in the morning. That type of answer has been deeply trained into the human networks and also requires no deep insight.

But if a human is allowed time and internal reasoning iterations, so should the LLM when determining if it has deep insight. Right now we're simply observing input -> output of LLMs, the equivalent of snap answers from a human. But nothing says it couldn't instead be an input -> extensive internal dialogue, maybe even between multiple expert models for seconds, minutes or hours, that are not at all visible to the prompter -> final insightful answer. Maybe future LLMs will say, "let me get back to you on that".

Re:Large language models lack deep insights or a theory of mind by lukev on Thursday November 30, 2023 @03:20PM (#38478545)
Completely agree.

From a computer science point of view: a single prompt/response cycle from a LLM is equivalent to a pure function; the answer is a function of the prompt and the model weights and is fundamentally reducible to solving a big math equation (in which each model parameter is a term.)

It seems almost self evident that "reasoning" worthy of the name would involve some sort of iterative/recursive search process, invoking the model and storing/reflecting/improving on answers methodically.

There's been a lot of movement in this direction with tree-of-thought/chain-of-thought/graph-of-thought prompting, and I would bet that if/when we get AGI, it's a result of getting the right recursive prompting pattern + retrieval patterns + ensemble models figured out, not just making ever-more-powerful transformer models (thought that would certainly play a role too.)

The LLM isn't the whole brain. Just the area responsible for language and cultural memory.

Re:Large language models lack deep insights or a theory of mind by kaba0 on Thursday November 30, 2023 @03:49PM (#38478847)
Ideally a recursive execution would also be a pure function - maybe a better way to put it about current LLMs is that they are a single mathematical expression being built up from a fix number of nodes and only addition and multiplication.
Re:Large language models lack deep insights or a theory of mind by lukev on Thursday November 30, 2023 @04:16PM (#38479239)
yes, except the "reasoning" process should also be able to look up facts (retrieval) and invoke external tools, making it non-pure.
Re:Large language models lack deep insights or a theory of mind by Scubabear68 on Thursday November 30, 2023 @07:40PM (#38481395)
Agree. I am not an expert on AGI, but I suspect this route is a dead end because I believe true AGI requires interaction with an external environment to learn eg there needs to be a feedback loop to external things that it can interact with and learn from.

“You learn best from your mistakes” comes to mind here.

And not just shallow feedback loops but bigger, more profound ones eg “I didn’t go to work for a month and so I was fired and and now I have no money and it will now be hard to be hired again”.

Re:Large language models lack deep insights or a theory of mind by hskalin on Friday December 01, 2023 @01:42PM (#38490529)
Do you have any good resources for chain-of-thought prompting experiments?
Re:Large language models lack deep insights or a theory of mind by I_Am_Nous on Thursday November 30, 2023 @02:53PM (#38478251)
This reminds me of The Last Question by Isaac Asimov. I also think if we stopped expecting all LLMs to have an immediate answer, it would be relatively easy to shim some kind of "conscience" to direct the output in different ways. Similar to the safeties already in place in LLMs, but instead of it just saying "NO DON'T SAY THAT" it can dialog internally to change what the output is until it reaches what it believes to be the agreed upon best answer.
Re:Large language models lack deep insights or a theory of mind by lifeisstillgood on Thursday November 30, 2023 @02:59PM (#38478305)
It would have an emotional reaction to certain "thought constructs" and would be guided by that.

Or we could just give them three laws

Re:Large language models lack deep insights or a theory of mind by I_Am_Nous on Thursday November 30, 2023 @03:21PM (#38478549)
With current LLMs the three laws might be tough to implement in a way that can't be prompt injected around. That's why I described the extra bits as a "conscience" which could enforce the three laws. Maybe the three laws are the internal conscience's context prompt while the main LLM is more able to think anything in general and then the output is tuned down by the conscience?

Otherwise the laws will have to be implemented as weights or during training so the model explicitly knows the laws and would never even be capable of doing something against them.

Re:Large language models lack deep insights or a theory of mind by pixl97 on Thursday November 30, 2023 @03:48PM (#38478842)
I mean, the whole purpose of the I, Robot story was to show you that the 3 laws didn't work. We had the first story on prompt injection decades ago and we just didn't realize it.
Re:Large language models lack deep insights or a theory of mind by ShamelessC on Thursday November 30, 2023 @07:16PM (#38481179)
Indeed. Asimov says as much in interviews - the laws are flawed from the get go. It is the naive belief that the laws work by various scientists that creates these Sherlock-style mysteries/crimes to be solved in the first place.

I suppose the real wisdom there is that humans are doomed to fail at alignment if we create sentience and expect it only to serve us.

Re:Large language models lack deep insights or a theory of mind by TeMPOraL on Thursday November 30, 2023 @05:03PM (#38479832)
Obligatory reminder that the "three laws" were invented to be deconstructed, with Asimov spending a lot of pages showing many ways in which they completely fail, illustrating that AI alignment is a hard problem.
Re:Large language models lack deep insights or a theory of mind by mr_toad on Thursday November 30, 2023 @05:12PM (#38479921)
> I also think if we stopped expecting all LLMs to have an immediate answer, it would be relatively easy to shim some kind of "conscience" to direct the output in different ways.

If the shim was just another AI, then how do you align that AI? Who watches the watchers? But if it was a deterministic algorithm it would probably fail for the same reasons that algorithmic AI never went anywhere.

Re:Large language models lack deep insights or a theory of mind by I_Am_Nous on Thursday November 30, 2023 @05:47PM (#38480322)
A great point! A smaller AI with a rather limited parameter count could be trained for individual needs so some things (chat moderation) might be easier to do than other things (fact check peer reviewed papers in a verifiable way). For some use cases it would be overkill to have a conscience but an AI spokesperson for a company will probably have a company-aligned conscience for obvious reasons.
Re:Large language models lack deep insights or a theory of mind by jvanderbot on Thursday November 30, 2023 @05:41PM (#38480233)
I think on of the reasons we require so much data is that we try to bake all that "simulated experience and internal dialogue" into the snap responses. I bet if you could do an efficient sim/test retraining, you'd do data-driven responses on the fly.
Re:Large language models lack deep insights or a theory of mind by two_in_one on Thursday November 30, 2023 @02:43PM (#38478131)
Just note that loop doesn't have to be visible from outside. It can be internal, with another driving thread asking right questions. Inner monologue. Then the summary is given back to user. This will give the model space for 'thinking' with internally generated text much large than the visible prompt + output. This way multi-step logic can be implemented.
Re:Large language models lack deep insights or a theory of mind by chii on Thursday November 30, 2023 @07:46PM (#38481444)
> with another driving thread asking right questions. Inner monologue.

spoilers warning:

Is that basically the plot to westworld?

Re:Large language models lack deep insights or a theory of mind by two_in_one on Thursday November 30, 2023 @09:06PM (#38482116)
Inner monologue idea was around for a while. That's interesting that it actually materializes. It was thought that AGI will be operating with some abstracts, logic rules, probability calculations, etc. Not thinking in plain English.
Re:Large language models lack deep insights or a theory of mind by lixy on Thursday November 30, 2023 @02:47PM (#38478183)
Yep, prototype exactly that this past week. With a strong instruction spec prompt from the start, you can have an AI come up with a much better answer by making sure it knows it has time to answer the questions and how it should approach the problem in stages.

The great part is with clear enough directions it also knows how to evaluate whether its done or not.

Re:Large language models lack deep insights or a theory of mind by hskalin on Friday December 01, 2023 @01:52PM (#38490642)
Can you share the kind of prompts you used?
Re:Large language models lack deep insights or a theory of mind by twobitshifter on Thursday November 30, 2023 @03:42PM (#38478774)
There are plenty of models that use introspection and check answers, that’s the idea behind let’s think about it step by step.
Re:Large language models lack deep insights or a theory of mind by tinco on Thursday November 30, 2023 @03:46PM (#38478817)
I feel the "let's think about it step by step" is a bit of a hack. To circumvent the fact that there's no external loop you use the fact that it gets re-run on every token so you can store a bit of state in the tokens that it's already generated.

Or am I misunderstanding something about that technique?

Re:Large language models lack deep insights or a theory of mind by twobitshifter on Thursday November 30, 2023 @04:02PM (#38479013)
You are right, it’s sometimes called zero shot chain of thought, but it’s a way of getting the type of thing you are describing to happen. The LLMs somehow process things in a perceived step by step to get a much improved answer. Whether the external loop or an llm imposed internal loop, does it matter? Are our own minds looping or just adding tokens?
Re:Large language models lack deep insights or a theory of mind by tinco on Thursday November 30, 2023 @04:45PM (#38479641)
Yeah that's true, and I do believe there's a good chance our minds are perpetually adding tokens. But our minds also have an efficient/effective way of dealing with the context cut off. We don't have (or we don't experience) a hard cut off of our memory context. Instead the tokens are increasingly lossily compressed as they age out of our memory, the lossiness amount being based both on time passed but also on some fancy value function. And that combined with a "system" (or trained/fine-tuned in) prompt that motivates the LLM to reason in a way that is conducive to working with that kind of memory would be a sort of single-shot AGI system. Where single-shot is lying a bit because it's just infinitely looping.

I guess from that perspective it might make sense to test if such a thing is already happening in current LLM's and my dismissive attitude stems from the fact that I've played with them enough to know that they currently don't.

Re:Large language models lack deep insights or a theory of mind by vidarh on Thursday November 30, 2023 @05:30PM (#38480113)
The main thing that may matter, at least in the short run, is that an external loop allows us to inject additional steps by applying heuristics and allowing tool use. E.g. we can let the LLM "realise" there errors in its code and have it continue from an injected "thought" about making sure to fix the errors from the compiler before presenting it's output, or "remembering" that it needs test cases etc.

We can also potentially add longer term memory - summarise the context, and judge which parts are important and stuff them in a vector store, and now and again swap in similar pieces of past context.

But of course it's not either or - better prompting to get the LLMs to do better from the start doesn't compete with then feeding that into an external loop as well.

Re:Large language models lack deep insights or a theory of mind by uoaei on Thursday November 30, 2023 @12:55PM (#38476605)
> They're just system 1 thinking, the equivalent of putting a gun to someone's head and asking them to solve a large division in 2 seconds.

No, it's the equivalent of putting a gun to someone's head and asking them "what are my intentions?" Which is readily available to any being with a theory of mind.

Re:Large language models lack deep insights or a theory of mind by choudharism on Thursday November 30, 2023 @01:05PM (#38476739)
I don’t think LLMs have theory of mind, but your point is not very strong. You can literally query ChatGPT right now and see that it can figure out intentions (both superficial and deep) of a gun is held to a head quite easily.

Because, obviously, training data probably includes a decent amount of motivation breakdowns as a function of coercion.

It doesn’t know why, but it knows what to say.

Re:Large language models lack deep insights or a theory of mind by og_kalu on Thursday November 30, 2023 @12:57PM (#38476633)
LLMs don't fail those kind of tasks though. 4 is very good at keeping track of who knows what and why in a story. You can test this yourself.
Re:Large language models lack deep insights or a theory of mind by bongodongobob on Thursday November 30, 2023 @03:40PM (#38478752)
What? My answer would be "I have no idea!"

Are they faking it? Will they murder me? Are they just trying to scare me?

I don't understand the point you're trying to make.

Re:Large language models lack deep insights or a theory of mind by FrustratedMonky on Thursday November 30, 2023 @01:35PM (#38477202)
Don't think so.

Put gun to persons ahead.

Ask them to do a division.

Then screaming at them "HOW DID YOU DO THAT, TELL ME NOW, OR YOU'RE TOAST".

Even most humans would splutter and not be able to answer.

Re:Large language models lack deep insights or a theory of mind by uoaei on Thursday November 30, 2023 @01:58PM (#38477539)
The division problem is arbitrary and irrelevant to any notion of theory of mind. If you have a better example, you can feel free to offer one. I already did so above.
Re:Large language models lack deep insights or a theory of mind by FrustratedMonky on Thursday November 30, 2023 @04:38PM (#38479556)
The point is not about the example of division, it is about 'tell me your intentions'.

A human also can't explain how their neurons calculated a division, or anything else, or 'intention', even if a gun is pointed at them.

So, why is that a fault of AI if it can't explain how itself is working? It is not a requirement for AGI.

Re:Large language models lack deep insights or a theory of mind by uoaei on Thursday November 30, 2023 @05:48PM (#38480348)
Every point you made here either reiterates something I've already said or misinterprets the nature of the discussion altogether.
Re:Large language models lack deep insights or a theory of mind by FrustratedMonky on Friday December 01, 2023 @10:51AM (#38488165)
Maybe,

you :"Which is readily available to any being with a theory of mind."

I'm saying humans do not have this ability. We don't have a valid proven theory of mind, and even if we did, that knowledge would not change how our brains process inputs, and somehow give us access to how we processed an answer. We don't know our 'intention'

Re:Large language models lack deep insights or a theory of mind by uoaei on Friday December 01, 2023 @12:50PM (#38489745)
I see the confusion now. Theory of mind is about recognizing the existence of other minds, not about being able to plumb the depths of one's own mind. If anyone had the latter skill, psychotherapy wouldn't need to exist and no one would ever have heard of Freud.
Re:Large language models lack deep insights or a theory of mind by rf15 on Thursday November 30, 2023 @01:00PM (#38476670)
they can't reason though, sadly - the premise does not hold.
Re:Large language models lack deep insights or a theory of mind by FrustratedMonky on Thursday November 30, 2023 @01:37PM (#38477228)
So your premise is correct? Please back up the opposite. If you can, you should publish.

These responses are logically the same as "No You".

Re:Large language models lack deep insights or a theory of mind by TeMPOraL on Thursday November 30, 2023 @05:07PM (#38479874)
That's literally GP's point though - they don't reason because reasoning requires a loop, which you deny them and then say "see, it can't reason".
Re:Large language models lack deep insights or a theory of mind by creer on Thursday November 30, 2023 @11:06PM (#38483010)
At the same time, if LLMs are based on all or enough human writings then don't they necessarily contain a theory of mind? A rather general, smoothed out and still neurotic one probably. But still just like an LLM can't be expected to have a specific knowledge of hydraulics, it also has read more hydraulics than even experts might be expected to. That's the entire issue about it, right? This issue of "is most of our mind basically just mixing and matching stuff we have seen, read, heard?" Do humans have some magical theory of mind that somehow stands ASIDE from all the "normal" learned stuff?

Of course, yes, we do know one thing that's missing in LLMs, which is "loop and helpers" like you describe. Which I'm sure many people are currently hacking at - one way being for the LLM to talk to itself.

But as for "a theory of mind", if enough writings served as input, then LLMs do have plenty of that.

Another question is whether LLMs are raised to behave like humans (which might be where they most NEED some theory of mind). Of course not. The ones we know most about are only question answerers. The theory of mind they might have (that is not negated by the lack of loop and internal deliberation) may be overwhelmed by the pre- and post-processing: "no sex, no murder plots, talk to the human like they are 5, bla bla bla". And yet you can ask things like "Tell it like you are speaking to 5 year olds who want to have a fun time". Some theory of mind makes it through.

Re:Large language models lack deep insights or a theory of mind by neuronerdgirl on Friday December 01, 2023 @10:21AM (#38487734)
That's not what theory of mind is. Theory of mind is the understanding - not academic understanding, but functional, meaningful understanding that guides your behavior - that the other people (AIs?) around you have their own separate thoughts and feelings from your own. This is what leads to the stark behavioral differences as a child develops from ~3-6 years of age. Imagine a child is playing with their toys and puts one of them in a box out of the room, then later asks you to retrieve it for them. A 3 year old will assume that, because they know they put it in the box, that you know it is there as well. (Incidentally, this is where a lot of tantrums come from, because they can't understand why you aren't reading their mind and get frustrated.) A 6 year old will not make this assumption.

While this is a simple example, the concept extrapolates to more complex issues as we develop cognitively and emotionally. Not only is theory of mind about recognizing that your thoughts and feelings are separate and distinct, but also being able to project your understanding into others to predict how they might be thinking and feeling in their own separate circumstance. This is the fundamental basis of empathy, or being able to predict and understand how others around you may be feeling given their unique current circumstances even if they are different from your own.

LLMs have not demonstrated any of the above understanding. Again, I'm sure they could generate a definition for it, and on any given prompt they could probably even generate some generic-ish text that could pose as empathy, but so can a horoscope. But there is not the deep, continuous comprehension of the user as a separate thinking and feeling agent that is fundamental to the idea of theory of mind.

Re:Large language models lack deep insights or a theory of mind by creer on Friday December 01, 2023 @05:48PM (#38493629)
I think this branch of the discussion was granting an "of course, for the moment" on this because of the lack of loop / internal conversation.

After that, the entire field of AI is about making bits and register transfers - a written form of knowledge - into something indistinguishable from the wet version. Making a priori and irreconcilably different "academic understanding" and "functional, meaningful understanding" is going to be a problem.