The correct response rate chart doesn't even use the results from the concise prompt.
I have a blog post in coming on this topic, but yes, this is right.
My method is to first get the LLM to answer the question, and THEN feed the answer back the LLM to extract the answer using constraints + grammar/logit bias/regex to parse the answer. Previously, I constrained to a single true/false token, which worked, but fails on complex queries.
So I split the decision making into a "justification" portion[0], and a "parsing" portion. I found that even crafting the prompt matters here, if you start with or end with, "It's very important to the response includes 'The answer is:'", then the model will lead with that response or only reply with that response. So I put it in the middle of the prompt, and end with with a request to justify the response. As a result, most models will reason their way to the answer, and then end with 'The answer is:'.
https://github.com/ShelbyJenkins/llm_client/blob/e3c4a860dda...
It was still incorrect, but when asked to show its working it formatted the answer prettily.
https://chatgpt.com/share/dcb4ff4e-e8a2-463b-86ec-9caf10b6e6...
Sometimes they get the answer right to something really complex because it fits a pattern, but sometimes they answer with something really really stupid.
I’m guessing you are in denial that we can make a simulated reasoning machine?
We haven't even started to approach the largest problem which is moving beyond what is essentially a greedy token level search of this linguistic space. That is, we can't really pick an output that maximized the likelihood of the entire sequence, rather we're simply maximizing the likelihood of each part of the sequence.
LLMs are not reasoning machines. They are basically semantic compression machines with a build in search feature.
Can you compress for me Van Gogh's Starry Night, please? I'd like to send a copy to my dear old mother who has never seen it. Please make sure when she decompresses the picture she misses none of the exquisite detail in that famous painting.
I see from your profile you are focused on your own personal and narrow definition of reasoning. But I’d argue there is a much broader and simpler definition. Can you summarize and apply learnings. This can.
That's important to understand. What I have in my profile is not some idiosyncratic idea about reasoning, it's the standard, formal understanding of what reasoning means, as it has developed in practice, in AI research in the last many decades.
I appreciate that there are many people who opine about reasoning who are not aware of that prior work and come up with their own ideas about what "reasoning" means, and some are even AI researches which is very concerning but I can't do anything about that except push back against such uninformed opinions.
>> This can.
I'm sorry, what can?
But keep lecturing everyone.
The only issue is ambiguity. What can be generated strongly depends on the order of the tokens. A slight variation can change the meaning and the result is worthless. Understanding is the guardrail against meaningless statement and LLMs lack it.
Also the above description is reductive to the point of "Cars can't get you anywhere because they aren't horses."
This is just a god of the gaps argument. Understanding is a form of semantic compression. So you're saying we have a system that can learn and construct a database of semantic information, then search it and compose novel, structured and coherent semantic content to respond to an a priori unknown prompt. Sounds like a form of reasoning to me. Maybe it's a limited deeply flawed type of reasoning, not that human reason is perfect, but that doesn't support your contention that it's not reasoning at all.
Sophisticated folks aren't doing the worlds most simplistic/stupid decoding.
Gotta go beyond LLMs 101 to see what's actually happening. Even in training folks are predicting several tokens ahead.
The problem is that these models weren't trained to reason. For the task of reasoning, they are overfitting to the dataset. If you want a machine to reason, then build and train it to reason, don't train it to do something else and then expect it to do the thing you didn't train it for.
Except they kind of were. Specifically, they were trained to predict next tokens based on text input, with the optimization function being, does the result make sense to a human?. That's embedded in the training data: it's not random strings, it's output of human reasoning, both basic and sophisticated. That's also what RLHF selects for later on. The models are indeed forced to simulate reasoning.
> don't train it to do something else and then expect it to do the thing you didn't train it for.
That's the difference between AGI and specialized AI - AGI is supposed to do the things you didn't train it to do.
If we tested humans on first thought questions and answers in 5 seconds or less on half the problems we did on LLMs — we might prove humans can’t reason as well
some people actually try, and see that LLMs are not there yet
A simulated reasoning machine being possible does not mean that current LLMs are simulated thinking machines.
Maybe you should try asking chatgpt for advice on how to understand other people’s perspectives: https://chatgpt.com/share/3d63c646-859b-4903-897e-9a0cb7e47b...
Obviously that was implied in my statement. Dude we aren’t all 4 year olds that need a self righteous lesson
What was implied by your statement? That you don’t understand other people’s perspectives?
If it really needs to do this 'thinking out loud', could it do that under the hood and not in the final output on my screen? Its first pass could use as many words as it wants to compute the answer, but once the answer is computed please go back and make it short.
Not to take away from your point that maybe the prompt is the problem in these reasoning questions.
I'm fairly certain we'll soon realize that what's happening here is that the markov chain being run over latent space needs a certain amount of "warmup" before it starts sampling from the optimal region. HMC samplers for Bayesian methods have this same property.
The terms "reasoning", "computing" or "thinking" for this stage should be considered metaphors rather than explanations for what's happening, which is really waiting for a random walk to start sampling from the typical-set.
Sometimes I think I'd prefer it to "think" before answering anyhow. The immediate thinking out loud text can be irritating for some irrational reason.
You're still paying for those thinking tokens, or at the very least have to wait for them to be generated.
You can always make excuses for the LLM afterwards, but software with hidden risks like this would not be considered good or reliable in any other context.
I once gave a 10-dollar bill to a young man serving at the cashier at a store, and he gave me 14 dollars back as a change. I pointed out that this made no sense. He bent down, looked closer at the screen of his machine, and said "Nope, 14 dollars, no mistake". I asked him if he thought I gave him 20. He said no, and even shown me the 10-dollar bill I just gave him. At that point I just gave up and took the money.
Now that I think about it, there was an eerie similarity between this conversation and some of the dialogues I had with LLMs...
Good example: I sunk probably an hour into trying to get Gemini Advanced to help me integrate it with a personal Google Calendar account. I kept asking it things and going crazy because nothing lined up with the way things worked. Finally, it referred to itself as Bard and I realized it was giving me information for a different product. As soon as I asked "are you giving me instructions for Gemini Advanced or Bard?" it was like "OH LOL WOOPS!! YOU GOT ME BRO! XD I CAN'T DO ANY OF THAT! LOL." Which, honestly, is great. Being able to evaluate its answers to realize it's wrong is really neat. Unfortunately, it was neat too late and too manually to stop me from wasting a ton of time.
I have decades of experience working in software-- imagine some rando that didn't know what the hell Bard was or even imagine this thing with "Advanced" in the name couldn't even distinguish between its own and other products' documentation.
Did it evaluate its answers, or did your expression of doubt cause the eager-to-please language model to switch from "generate (wrong) instructions because that's what the user asked for" to "acknowledge an error because that's what the user asked for"?
How many times have we seen "Oops, you're right! 2 + 2 is actually 5! I apologize for saying it was 4 earlier!"
If you're among technologists discussing LLMs academically, as we are, that's a reasonable approach. However, I see a lot of people fail to distinguish that from LLM-powerd products sold to the general public as intelligent bots that can understand your plain english and output answers.
People use their existing mental models when interacting with something. If you have 3 different interfaces with a widget to trigger the same exact function, but one look like a music play button, one looks like a gas pedal, and one looks like mechanical pinball plunger, we interact with those things differently because we know how those things work. In this context, chatbots are designed to engage people's existing mental model for chatting with a person via text. The further you stray from people's expectations of human chat, the further you are from people's expectations, for better, or worse.
If you're selling someone a product claiming it understands plain language questions and gives plain language answers, then not getting the right answer to that question makes it idiotic. The subtleties aren't within most users' grasp, and the "FYI: this thing might be full of shit" disclaimer isn't helpful if you don't know enough about what you're asking to administer a proper smell test.
Your statements are obviously not wrong, but I see people saying these things like its reasonable for non-technical end users to reason about those subtleties. Considering how those things are marketed, I really don't think it is.
Yes this is a common thing I see people who think LLMs are idiots do.
The more an LLM talks the smarter it gets _because that's the only way it can compute anything_. Imagine saying that Turing machines fail the Church–Turing thesis because they can't solve 3-sat for N variables in N moves or less.
That's what you're doing to an LLM when you ask it to be concise.