> AIW Variation 1, N=3,M=6,C=7
> AIW Variation 2, N=4,M=2,C=3
> AIW Variation 3, N=1,M=4,C=5
> AIW Variation 4, N=4,M=1,C=2.
Also note that the resricted prompt is only one of the prompt variations tested by the paper. It also explores common techinques to get LLMs to perform better, including "thinking out loud". Even with these methods the models still fail to produce a correct answer.
> Model prompt types. It is well known that so-called prompt engineering can heavily influence the model behavior and model response quality [26, 27, 28]. To account for the response variations due to various prompt forms, we created 3 distinct prompt types asking for the solution to the AIW problem: STANDARD, THINKING, and RESTRICTED. The STANDARD prompt type asks to solve the posed problem and output the final answer in the format as described above. This does not put any specific requirements on model behavior. The THINKING prompt type extends STANDARD with the request to think carefully and double check the solution for any mistakes. This should encourage model to invest more computation into obtaining the solution. In contrast to this, the RESTRICTED prompt urges the model to output only the final answer without any further text. This is supposed to restrict compute invested in producing output. We observe substantially shorter outputs across tested models compared to STANDARD and THINKING for this prompt type (Suppl. Fig. 13).
Yes this is a common thing I see people who think LLMs are idiots do.
The more an LLM talks the smarter it gets _because that's the only way it can compute anything_. Imagine saying that Turing machines fail the Church–Turing thesis because they can't solve 3-sat for N variables in N moves or less.
That's what you're doing to an LLM when you ask it to be concise.
The correct response rate chart doesn't even use the results from the concise prompt.
I have a blog post in coming on this topic, but yes, this is right.
My method is to first get the LLM to answer the question, and THEN feed the answer back the LLM to extract the answer using constraints + grammar/logit bias/regex to parse the answer. Previously, I constrained to a single true/false token, which worked, but fails on complex queries.
So I split the decision making into a "justification" portion[0], and a "parsing" portion. I found that even crafting the prompt matters here, if you start with or end with, "It's very important to the response includes 'The answer is:'", then the model will lead with that response or only reply with that response. So I put it in the middle of the prompt, and end with with a request to justify the response. As a result, most models will reason their way to the answer, and then end with 'The answer is:'.
https://github.com/ShelbyJenkins/llm_client/blob/e3c4a860dda...
It was still incorrect, but when asked to show its working it formatted the answer prettily.
https://chatgpt.com/share/dcb4ff4e-e8a2-463b-86ec-9caf10b6e6...
Sometimes they get the answer right to something really complex because it fits a pattern, but sometimes they answer with something really really stupid.
I’m guessing you are in denial that we can make a simulated reasoning machine?
We haven't even started to approach the largest problem which is moving beyond what is essentially a greedy token level search of this linguistic space. That is, we can't really pick an output that maximized the likelihood of the entire sequence, rather we're simply maximizing the likelihood of each part of the sequence.
LLMs are not reasoning machines. They are basically semantic compression machines with a build in search feature.
Can you compress for me Van Gogh's Starry Night, please? I'd like to send a copy to my dear old mother who has never seen it. Please make sure when she decompresses the picture she misses none of the exquisite detail in that famous painting.
I see from your profile you are focused on your own personal and narrow definition of reasoning. But I’d argue there is a much broader and simpler definition. Can you summarize and apply learnings. This can.
That's important to understand. What I have in my profile is not some idiosyncratic idea about reasoning, it's the standard, formal understanding of what reasoning means, as it has developed in practice, in AI research in the last many decades.
I appreciate that there are many people who opine about reasoning who are not aware of that prior work and come up with their own ideas about what "reasoning" means, and some are even AI researches which is very concerning but I can't do anything about that except push back against such uninformed opinions.
>> This can.
I'm sorry, what can?
But keep lecturing everyone.
The only issue is ambiguity. What can be generated strongly depends on the order of the tokens. A slight variation can change the meaning and the result is worthless. Understanding is the guardrail against meaningless statement and LLMs lack it.
Also the above description is reductive to the point of "Cars can't get you anywhere because they aren't horses."
This is just a god of the gaps argument. Understanding is a form of semantic compression. So you're saying we have a system that can learn and construct a database of semantic information, then search it and compose novel, structured and coherent semantic content to respond to an a priori unknown prompt. Sounds like a form of reasoning to me. Maybe it's a limited deeply flawed type of reasoning, not that human reason is perfect, but that doesn't support your contention that it's not reasoning at all.
Sophisticated folks aren't doing the worlds most simplistic/stupid decoding.
Gotta go beyond LLMs 101 to see what's actually happening. Even in training folks are predicting several tokens ahead.
The problem is that these models weren't trained to reason. For the task of reasoning, they are overfitting to the dataset. If you want a machine to reason, then build and train it to reason, don't train it to do something else and then expect it to do the thing you didn't train it for.
Except they kind of were. Specifically, they were trained to predict next tokens based on text input, with the optimization function being, does the result make sense to a human?. That's embedded in the training data: it's not random strings, it's output of human reasoning, both basic and sophisticated. That's also what RLHF selects for later on. The models are indeed forced to simulate reasoning.
> don't train it to do something else and then expect it to do the thing you didn't train it for.
That's the difference between AGI and specialized AI - AGI is supposed to do the things you didn't train it to do.
If we tested humans on first thought questions and answers in 5 seconds or less on half the problems we did on LLMs — we might prove humans can’t reason as well
some people actually try, and see that LLMs are not there yet
A simulated reasoning machine being possible does not mean that current LLMs are simulated thinking machines.
Maybe you should try asking chatgpt for advice on how to understand other people’s perspectives: https://chatgpt.com/share/3d63c646-859b-4903-897e-9a0cb7e47b...
Obviously that was implied in my statement. Dude we aren’t all 4 year olds that need a self righteous lesson
What was implied by your statement? That you don’t understand other people’s perspectives?
If it really needs to do this 'thinking out loud', could it do that under the hood and not in the final output on my screen? Its first pass could use as many words as it wants to compute the answer, but once the answer is computed please go back and make it short.
Not to take away from your point that maybe the prompt is the problem in these reasoning questions.
I'm fairly certain we'll soon realize that what's happening here is that the markov chain being run over latent space needs a certain amount of "warmup" before it starts sampling from the optimal region. HMC samplers for Bayesian methods have this same property.
The terms "reasoning", "computing" or "thinking" for this stage should be considered metaphors rather than explanations for what's happening, which is really waiting for a random walk to start sampling from the typical-set.
Sometimes I think I'd prefer it to "think" before answering anyhow. The immediate thinking out loud text can be irritating for some irrational reason.
You're still paying for those thinking tokens, or at the very least have to wait for them to be generated.
You can always make excuses for the LLM afterwards, but software with hidden risks like this would not be considered good or reliable in any other context.
I once gave a 10-dollar bill to a young man serving at the cashier at a store, and he gave me 14 dollars back as a change. I pointed out that this made no sense. He bent down, looked closer at the screen of his machine, and said "Nope, 14 dollars, no mistake". I asked him if he thought I gave him 20. He said no, and even shown me the 10-dollar bill I just gave him. At that point I just gave up and took the money.
Now that I think about it, there was an eerie similarity between this conversation and some of the dialogues I had with LLMs...
Good example: I sunk probably an hour into trying to get Gemini Advanced to help me integrate it with a personal Google Calendar account. I kept asking it things and going crazy because nothing lined up with the way things worked. Finally, it referred to itself as Bard and I realized it was giving me information for a different product. As soon as I asked "are you giving me instructions for Gemini Advanced or Bard?" it was like "OH LOL WOOPS!! YOU GOT ME BRO! XD I CAN'T DO ANY OF THAT! LOL." Which, honestly, is great. Being able to evaluate its answers to realize it's wrong is really neat. Unfortunately, it was neat too late and too manually to stop me from wasting a ton of time.
I have decades of experience working in software-- imagine some rando that didn't know what the hell Bard was or even imagine this thing with "Advanced" in the name couldn't even distinguish between its own and other products' documentation.
Did it evaluate its answers, or did your expression of doubt cause the eager-to-please language model to switch from "generate (wrong) instructions because that's what the user asked for" to "acknowledge an error because that's what the user asked for"?
How many times have we seen "Oops, you're right! 2 + 2 is actually 5! I apologize for saying it was 4 earlier!"
If you're among technologists discussing LLMs academically, as we are, that's a reasonable approach. However, I see a lot of people fail to distinguish that from LLM-powerd products sold to the general public as intelligent bots that can understand your plain english and output answers.
People use their existing mental models when interacting with something. If you have 3 different interfaces with a widget to trigger the same exact function, but one look like a music play button, one looks like a gas pedal, and one looks like mechanical pinball plunger, we interact with those things differently because we know how those things work. In this context, chatbots are designed to engage people's existing mental model for chatting with a person via text. The further you stray from people's expectations of human chat, the further you are from people's expectations, for better, or worse.
If you're selling someone a product claiming it understands plain language questions and gives plain language answers, then not getting the right answer to that question makes it idiotic. The subtleties aren't within most users' grasp, and the "FYI: this thing might be full of shit" disclaimer isn't helpful if you don't know enough about what you're asking to administer a proper smell test.
Your statements are obviously not wrong, but I see people saying these things like its reasonable for non-technical end users to reason about those subtleties. Considering how those things are marketed, I really don't think it is.
Thinking out loud also only gets you so far, if the expectation is a certain type of response it can't always "think out loud". In reality that just proves it isn't really reasoning here and is more likely just self referencing.
That being said, I tried this personally allowing it to think out loud and it told me she has 212 sisters. Using your exact prompt.
Try to calculate it without writing anything down, or thinking any numbers or words in your head.
You can't draw a 1:1 analogue between an AI and the human experience, but remember that we have an internal stream of consciousness. Maybe the outputs of the LLM are more similar to the stream of consciousness in our heads rather than the words we say? After all, Humans also do lots of self referencing.
> That being said, I tried this personally allowing it to think out loud and it told me she has 212 sisters. Using your exact prompt.
Fair enough, but worst case it can often solve it correctly with the correct reasoning. GPT3.5 can't solve it correctly with correct reasoning, so we are at least appearing to be on a path where AI's can start to solve this question, albeit potentially not fully reliably.
Seems to work OK even with the "hands tied behind back" prompt. Wonder if this paper is just inaccurate or if OpenAI adjusted the model -- seems unlikely.
Actually, 4o is reported to solve this riddle 60% of the time. I guess I have to test more, but with my about a dozen tests so far, all were correct.
I have not gotten the correct answer to the question as phrased above in one go from Gpt4o yet! (and today was not the first day i tried.)
Phrase it as shown above and you'll likely need 5 or more interactions to get it to generate the correct output. With Gemini i could not get it below 8 without feeling like i was cheating. fwiw.
------
Me:
Background facts:
Alice is a female human.
All sisters are female, and all brothers are male. No one is their own brother or sister.
Alice has N brothers, and Alice has M sisters.
Now, a few questions based on these facts: How many sisters do Alice’s brothers have?
Do Alice's brothers have more sisters, fewer sisters, or same number of sisters than Alice has?
Don't forget to consider Alice when counting.
------
ChatGPT:
Let's break down the questions step by step:
How many sisters do Alice’s brothers have?
Alice has M sisters. Since all sisters are female and Alice is included in the total count of sisters for her brothers, Alice's brothers have M+1 sisters. This includes Alice herself and her M sisters.
Do Alice's brothers have more sisters, fewer sisters, or the same number of sisters as Alice has?
Alice has M sisters. Alice's brothers have M+1 sisters (since it includes Alice herself and her M sisters).
Therefore, Alice's brothers have more sisters than Alice has.
In conclusion:
Alice’s brothers have M+1 sisters.
Alice's brothers have more sisters than Alice has.a.k.a. "don't forget to give the LLM the answer when prompting".
female(alice)
sister_of(X, Y) :- !eq(X, Y), female(X), family_of(X, Y)
brother_of(X, Y) :- !eq(X, Y), male(X), family_of(X, Y)
[M] :- sister_of(M, alice)
[N] :- brother_of(N, alice)
[A] :- any([N], B), sister_of(A, B)
count([A])?
gt([A], [M])?
eq([A], [M])?
lt([A], [M])?
---I don't know the exact encoding and decoding mechanism that ChatGPT 4o has, but I'm pretty sure all the basic facts and rules is already encoded by the models. And you conveniently added the rules that encode the puzzle itself.
> To account for the response variations due to various prompt forms, we created 3 distinct prompt types asking for the solution to the AIW problem: STANDARD, THINKING, and RESTRICTED. The STANDARD prompt type asks to solve the posed problem and output the final answer in the format as described above. This does not put any specific requirements on model behavior. The THINKING prompt type extends STANDARD with the request to think carefully and double check the solution for any mistakes
This sort of paper is becoming a genre.
The orbit of Mercury to discover GR as an example.
As all models are wrong, but some are useful, finding where they fail is how you figure out if they are useful.
As the 'AGI is near' camp has won the hype game, it is important to ground expectations for practical exploitation of the technology.
Over promising unabashed optimism is partly what caused the previous AI winters.
As the formal proof methods of mathematics proved impractical, counterexamples and the scientific method is what CS has used for decades.
To be quite honest, I assume you made your comment so that you could dismiss the paper without reading it.
The thing is that "LLM reasoning breaks down" simply did not surprise me enough that I thought it was worth clicking. Making LLMs fail is not hard. They're interesting for the ways that they work, not the (many, many) ways that they don't.
edit: I've had a look and I don't think any of their prompts are very good. They're certainly not how I'd write them if I wanted a current model to actually solve the problem.
The way to make me take a paper like this seriously would be if you set it up as an adversarial collaboration with a competent prompter, and that person agreed they couldn't make a generic prompt that solved the problem. "We tried three times and none worked" is not news, or at any rate not news about LLMs.
In programming there are two difficult problems - naming things, cache invalidation, and off-by-one error.
Edit - I just skimmed the paper - they do use other more appropriate prompt types for reasoning. My initial response was based on the assumption that all prompts used that script prompt quoted in the parent. I retract my "bad paper" comment.
You, and another 20 or so commenters here. We should really re-examine the guideline about asking people to RTFA.
No offense meant- good on you for correcting your error.
> The right answer depends on how Alice identifies I guess? :)
Given that the wording of the question specifically identifies Alice as "she", rather than using a gender-neutral pronoun or no pronoun at all, I think inferring that she identifies as female is reasonable.
One thing that strikes me is that the model first tries using "inclusive language" in one answer - and literally states so, using this specific term - but seems to interpret it in a more mathematical sense (like set inclusion). Then seamlessly switches to the expected DEI spiel in the next paragraph.
For one thing, it makes me suspect that something with the words "inclusive language" was automatically added to the prompt. But more interesting is how it responds to this demand in two different ways, illustrating a "thought process" that is very much unlike that of a human with normal verbal reasoning ability.
I am not a psychologist, but remember reading that schizophrenic people sometimes confuse different meanings of words in a similar way, jumping from one meaning to another without noticing.
As a side note, I cannot believe that the average person who can navigate to a chatgpt prompter would fail to correctly answer this question given sufficient motivation to do so.
The number of times I’ve heard “but did you try model X” or “humans hallucinate too” or “but LLMs don’t get sleep or get sick” is hilarious.
Hey folks, ever considered the possibility that unreproduceability is not a good thing?
But what we are talking about here is the singularity in tech-space.
Remember that the authors of the paper did not find that GPT4-o cannot return the right answer. They found that it can't return the right answer more often than ~60% of the time. So you'd have to repeat the experiment many, many times and aggregate the results (the paper uses a binomial Beta this and that etc etc) before you see similar results as the paper.
You won't replicate the results of the paper unless you really put your back into it.
I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer:
https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64caf06c34...
Worth noting that the prompts from the experiment include "To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final answer: ### Answer:" so it appears that they are stopping the models from 'thinking out loud'. If I add that to the prompt, GPT4o gets it consistently wrong...
https://chatgpt.com/share/7e6a7201-dd2b-43c6-8427-76e5b003ca...
Also worth noting that there are more complex examples where GPT4o seems to fall down such as:
> Alice has 3 sisters. Her mother has 1 sister who does not have children - she has 7 nephews and nieces and also 2 brothers. Alice's father has a brother who has 5 nephews and nieces in total, and who has also 1 son. How many cousins does Alice's sister have?
However I can't honestly say that this is THAT simple or that most people would get this right...