News for hackers. Stuff that matters.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by llm_trw on Wednesday June 05, 2024 @10:43AM (#40585414)
>Worth noting that the prompts from the experiment include "To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final answer: ### Answer:" so it appears that they are stopping the models from 'thinking out loud'. If I add that to the prompt, GPT4o gets it consistently wrong...

Yes this is a common thing I see people who think LLMs are idiots do.

The more an LLM talks the smarter it gets _because that's the only way it can compute anything_. Imagine saying that Turing machines fail the Church–Turing thesis because they can't solve 3-sat for N variables in N moves or less.

That's what you're doing to an LLM when you ask it to be concise.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by Miraste on Wednesday June 05, 2024 @11:12AM (#40585859)
As it says in the paper, they already did the research with less limiting prompts. All models were tested with another prompt that gave only the question and no further instruction, and a third prompt that asked the model to consider its answer carefully.

The correct response rate chart doesn't even use the results from the concise prompt.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by J_Shelby_J on Wednesday June 05, 2024 @11:54AM (#40586494)
> The more an LLM talks the smarter it gets

I have a blog post in coming on this topic, but yes, this is right.

My method is to first get the LLM to answer the question, and THEN feed the answer back the LLM to extract the answer using constraints + grammar/logit bias/regex to parse the answer. Previously, I constrained to a single true/false token, which worked, but fails on complex queries.

So I split the decision making into a "justification" portion[0], and a "parsing" portion. I found that even crafting the prompt matters here, if you start with or end with, "It's very important to the response includes 'The answer is:'", then the model will lead with that response or only reply with that response. So I put it in the middle of the prompt, and end with with a request to justify the response. As a result, most models will reason their way to the answer, and then end with 'The answer is:'.

https://github.com/ShelbyJenkins/llm_client/blob/e3c4a860dda...

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by muglug on Wednesday June 05, 2024 @11:47AM (#40586378)
I think you're wrong about that — I just tried prompting ChatGPT 4o to show all its working before giving an answer.

It was still incorrect, but when asked to show its working it formatted the answer prettily.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by pawelmurias on Wednesday June 05, 2024 @11:08AM (#40585801)
LLMs are idiots. They can't reason properly and only parrot stuff

https://chatgpt.com/share/dcb4ff4e-e8a2-463b-86ec-9caf10b6e6...

Sometimes they get the answer right to something really complex because it fits a pattern, but sometimes they answer with something really really stupid.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by tsunamifury on Wednesday June 05, 2024 @11:47AM (#40586376)
Why are so many people so insistent on saying this?

I’m guessing you are in denial that we can make a simulated reasoning machine?

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by PheonixPharts on Wednesday June 05, 2024 @11:54AM (#40586498)
People keep saying it because that's literally how LLMs work. They run Montecarlo sampling over a very impressive latent linguistic space. These models are not fundamentally different than the Markov chains of yore except that these latent representations are incredibly powerful.

We haven't even started to approach the largest problem which is moving beyond what is essentially a greedy token level search of this linguistic space. That is, we can't really pick an output that maximized the likelihood of the entire sequence, rather we're simply maximizing the likelihood of each part of the sequence.

LLMs are not reasoning machines. They are basically semantic compression machines with a build in search feature.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by pretendscholar on Wednesday June 05, 2024 @12:59PM (#40587415)
The best compression is some form of understanding
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by YeGoblynQueenne on Wednesday June 05, 2024 @04:07PM (#40589892)
That's a fascinating insight and it sound so true!

Can you compress for me Van Gogh's Starry Night, please? I'd like to send a copy to my dear old mother who has never seen it. Please make sure when she decompresses the picture she misses none of the exquisite detail in that famous painting.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by pretendscholar on Wednesday June 05, 2024 @04:33PM (#40590236)
Okay yes so not really having an artists vocabulary I couldn't compress it as well as someone who has a better understanding of Starry Night. An artist that understands what makes Starry Night great could create a work that evokes similar feelings and emotions. I know this because Van Gogh created many similar works playing with the same techniques, colors, and subjects such as Cypresses in Starry Night and Starry Night over the Rhone. He was clearly working from a concise set of ideas and techniques which I would argue is understanding/compression.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by YeGoblynQueenne on Wednesday June 05, 2024 @06:14PM (#40591208)
Fine, but we were talking about compression, not about imitation, or inspiration, and not about creating "a work that evokes similar feelings and emotions". If I compress an image, what I get when I decompress it is that image, not "feelings and emotions", yes? In fact, that's kind of the whole point: I can send an image over the web and the receiver can form their own feelings and emotions, without having to rely on mine.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by tsunamifury on Wednesday June 05, 2024 @06:34PM (#40591415)
Simple reasoning is a side effect of compression. That is all.

I see from your profile you are focused on your own personal and narrow definition of reasoning. But I’d argue there is a much broader and simpler definition. Can you summarize and apply learnings. This can.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by YeGoblynQueenne on Thursday June 06, 2024 @08:34AM (#40596661)
To clarify, what I have in my profile is not my "own personal" definition of reasoning. It's how reasoning is understood in computer science and AI, and I am an expert on the subject through my doctoral studies and my current post-doc research.

That's important to understand. What I have in my profile is not some idiosyncratic idea about reasoning, it's the standard, formal understanding of what reasoning means, as it has developed in practice, in AI research in the last many decades.

I appreciate that there are many people who opine about reasoning who are not aware of that prior work and come up with their own ideas about what "reasoning" means, and some are even AI researches which is very concerning but I can't do anything about that except push back against such uninformed opinions.

>> This can.

I'm sorry, what can?

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by tsunamifury on Thursday June 06, 2024 @11:52AM (#40598886)
Academics have gotten AI wrong since its inception and now are relegated to the trailing edges of the field.

But keep lecturing everyone.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by YeGoblynQueenne on Thursday June 06, 2024 @01:03PM (#40599767)
Unfortunately I'm used to people on the internet wearing their ignorance on their sleeve like a badge of honour and so I'm not surprised by the insults in your comment. Just a bit sad to be honest :(
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by skydhash on Wednesday June 05, 2024 @03:00PM (#40588925)
The best compression relies on understanding. What LLM is is mostly data how humans use words. We understand how to make this data (which is a compression of human text) and use it (generate something). AKA it’s “production rules”, but statistical.

The only issue is ambiguity. What can be generated strongly depends on the order of the tokens. A slight variation can change the meaning and the result is worthless. Understanding is the guardrail against meaningless statement and LLMs lack it.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by tsunamifury on Wednesday June 05, 2024 @03:08PM (#40589025)
You seem to entirely miss how attention layers work...
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by foobiekr on Wednesday June 05, 2024 @02:25PM (#40588501)
Exactly right and well said.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by tsunamifury on Wednesday June 05, 2024 @02:53PM (#40588832)
This type of self affirmation has a quality of denial.

Also the above description is reductive to the point of "Cars can't get you anywhere because they aren't horses."

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by naasking on Wednesday June 05, 2024 @04:43PM (#40590362)
> LLMs are not reasoning machines. They are basically semantic compression machines with a build in search feature.

This is just a god of the gaps argument. Understanding is a form of semantic compression. So you're saying we have a system that can learn and construct a database of semantic information, then search it and compose novel, structured and coherent semantic content to respond to an a priori unknown prompt. Sounds like a form of reasoning to me. Maybe it's a limited deeply flawed type of reasoning, not that human reason is perfect, but that doesn't support your contention that it's not reasoning at all.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by tsunamifury on Wednesday June 05, 2024 @05:23PM (#40590794)
It’s basically an argument that boils down to “it’s not because I don’t like it”
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by jononomo on Wednesday June 05, 2024 @05:27PM (#40590825)
I bite the bullet on the god of the gaps
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by danielmarkbruce on Wednesday June 05, 2024 @08:01PM (#40592004)
Beam search.

Sophisticated folks aren't doing the worlds most simplistic/stupid decoding.

Gotta go beyond LLMs 101 to see what's actually happening. Even in training folks are predicting several tokens ahead.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by Closi on Thursday June 06, 2024 @05:30AM (#40595365)
I don't think you can evaluate if an LLM is reasoning by looking purely at the mechanics, because if we looked inside a human brain we wouldn't be able to conclude that it can reason either (our test is 'i think therefore i am', not all these neurons look like they are plugged together in such a way that it enables reason).
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by camdenreslink on Wednesday June 05, 2024 @12:01PM (#40586597)
It is hard to trust any output from a machine that is confidently wrong so frequently. You need to already be knowledgable in a topic (or at least have a well attuned BS detector) to know if it is giving you correct responses. It can be a time saver and assistant in getting work done where you are already a subject matter expert, but it needs to get better to remove the human from the loop.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by hyperbovine on Wednesday June 05, 2024 @11:54AM (#40586496)
Because they understand how LLMs work. It's not reasoning. It's not simulating reasoning.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by imtringued on Wednesday June 05, 2024 @12:22PM (#40586860)
No it is because supervised and self supervised learning happen to produce reasoning as a byproduct. For some reason people think that telling a model to recite a trillion tokens somehow will improve it beyond the recitation of those tokens. I mean, in theory you can select the training data so that it will learn what you want, but then again you are limited to what you taught it directly.

The problem is that these models weren't trained to reason. For the task of reasoning, they are overfitting to the dataset. If you want a machine to reason, then build and train it to reason, don't train it to do something else and then expect it to do the thing you didn't train it for.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by TeMPOraL on Wednesday June 05, 2024 @05:53PM (#40591046)
> The problem is that these models weren't trained to reason.

Except they kind of were. Specifically, they were trained to predict next tokens based on text input, with the optimization function being, does the result make sense to a human?. That's embedded in the training data: it's not random strings, it's output of human reasoning, both basic and sophisticated. That's also what RLHF selects for later on. The models are indeed forced to simulate reasoning.

> don't train it to do something else and then expect it to do the thing you didn't train it for.

That's the difference between AGI and specialized AI - AGI is supposed to do the things you didn't train it to do.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by tsunamifury on Wednesday June 05, 2024 @07:02PM (#40591634)
I think people don’t recognize it’s currently doing single turn reasoning and demonstrating the building blocks of real time reasoning with continuous input.

If we tested humans on first thought questions and answers in 5 seconds or less on half the problems we did on LLMs — we might prove humans can’t reason as well

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by riku_iki on Wednesday June 05, 2024 @02:20PM (#40588430)
> I’m guessing you are in denial that we can make a simulated reasoning machine?

some people actually try, and see that LLMs are not there yet

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by TeMPOraL on Wednesday June 05, 2024 @05:40PM (#40590939)
There's some irony in seeing people parrot the argument that LLMs are parrots.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by tsunamifury on Wednesday June 05, 2024 @06:58PM (#40591611)
Also making errors in reasoning while saying LLM errors prove it can’t reason.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by elicksaur on Wednesday June 05, 2024 @06:13PM (#40591194)
Maybe people have different experiences with the products than you.

A simulated reasoning machine being possible does not mean that current LLMs are simulated thinking machines.

Maybe you should try asking chatgpt for advice on how to understand other people’s perspectives: https://chatgpt.com/share/3d63c646-859b-4903-897e-9a0cb7e47b...

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by tsunamifury on Wednesday June 05, 2024 @06:31PM (#40591386)
This is such a weirdly preachy and belligerent take.

Obviously that was implied in my statement. Dude we aren’t all 4 year olds that need a self righteous lesson

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by elicksaur on Wednesday June 05, 2024 @07:16PM (#40591744)
Weird to accuse a response of being belligerent when your initial comment stated that people who disagreed with you are in denial.

What was implied by your statement? That you don’t understand other people’s perspectives?

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by dematz on Wednesday June 05, 2024 @11:06AM (#40585775)
I often want chatgpt to answer concisely and tell it that.

If it really needs to do this 'thinking out loud', could it do that under the hood and not in the final output on my screen? Its first pass could use as many words as it wants to compute the answer, but once the answer is computed please go back and make it short.

Not to take away from your point that maybe the prompt is the problem in these reasoning questions.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by marzell on Wednesday June 05, 2024 @02:24PM (#40588476)
I believe the "thinking out loud" is fundamentally part of the process of "text completion" which is what it is doing. Certainly we can (and do) break things apart and add layers that could be used to effectively do this by adding more steps and processing time. But ultimately in a single turn, the entire conversation up to that point (including instructions you may have added telling it to not think out loud) is the input, and the output will reflect that.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by PheonixPharts on Wednesday June 05, 2024 @11:50AM (#40586436)
>_because that's the only way it can compute anything_

I'm fairly certain we'll soon realize that what's happening here is that the markov chain being run over latent space needs a certain amount of "warmup" before it starts sampling from the optimal region. HMC samplers for Bayesian methods have this same property.

The terms "reasoning", "computing" or "thinking" for this stage should be considered metaphors rather than explanations for what's happening, which is really waiting for a random walk to start sampling from the typical-set.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by sosuke on Wednesday June 05, 2024 @11:40AM (#40586266)
New option needed besides concise. Think quietly on your own, then answer concisely.

Sometimes I think I'd prefer it to "think" before answering anyhow. The immediate thinking out loud text can be irritating for some irrational reason.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by monsieurbanana on Wednesday June 05, 2024 @11:45AM (#40586336)
Just telling the llm to preface it's conclusion with "Answer:" in a parseable way would be better.

You're still paying for those thinking tokens, or at the very least have to wait for them to be generated.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by rossdavidh on Wednesday June 05, 2024 @11:05AM (#40585758)
Even if this were all true, it points to a fundamental risk of using LLM's for important tasks, which is that it is not at all clear to a user that this prompt would cause a problem. The LLM doesn't say "I'm sorry Dave, I just can't do that", it just complies with it and gets the wrong answer.

You can always make excuses for the LLM afterwards, but software with hidden risks like this would not be considered good or reliable in any other context.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by saurik on Wednesday June 05, 2024 @11:53AM (#40586480)
People really need to stop trying to model an LLM as some kind of magical software component: it all makes a lot more sense if you model it as an under-performing poorly-aligned employee; so like, maybe a distracted kid working for peanuts at your store. You wouldn't trust them to with all of your money and you wouldn't trust them to do a lot of math--if they had to be in charge of checkout, you'd make sure they are only given a point-of-sale terminal and their main job was to, at best, scan the barcodes and compare the total--and yet there are tasks you can imagine handing to them that you'd never give to a robot or computer even though they get it wrong a lot, as not all tasks need to be handled perfectly, they still understand extremely fuzzy tasks, and they are probably cheaper than a qualified adult (certainly cheaper than one who is being paid enough to "give a shit" and pay enough attention to not let you get robbed or even put themselves at some risk for you).
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by IlliOnato on Wednesday June 05, 2024 @02:39PM (#40588682)
You have a point...

I once gave a 10-dollar bill to a young man serving at the cashier at a store, and he gave me 14 dollars back as a change. I pointed out that this made no sense. He bent down, looked closer at the screen of his machine, and said "Nope, 14 dollars, no mistake". I asked him if he thought I gave him 20. He said no, and even shown me the 10-dollar bill I just gave him. At that point I just gave up and took the money.

Now that I think about it, there was an eerie similarity between this conversation and some of the dialogues I had with LLMs...

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by chefandy on Wednesday June 05, 2024 @12:34PM (#40587023)
While LLMs have incredible potential, and are even downright useful in their current format, they have the rather nasty tendency to confidently present bullshit that passes the smell test for most people. When it's wrong, it's not just wrong, by design, it's wrong but sounds plausible. Considering most people with subject matter expertise aren't going to consult such a bot for their own areas of expertise, that means most people will not be able to detect it intuitively.

Good example: I sunk probably an hour into trying to get Gemini Advanced to help me integrate it with a personal Google Calendar account. I kept asking it things and going crazy because nothing lined up with the way things worked. Finally, it referred to itself as Bard and I realized it was giving me information for a different product. As soon as I asked "are you giving me instructions for Gemini Advanced or Bard?" it was like "OH LOL WOOPS!! YOU GOT ME BRO! XD I CAN'T DO ANY OF THAT! LOL." Which, honestly, is great. Being able to evaluate its answers to realize it's wrong is really neat. Unfortunately, it was neat too late and too manually to stop me from wasting a ton of time.

I have decades of experience working in software-- imagine some rando that didn't know what the hell Bard was or even imagine this thing with "Advanced" in the name couldn't even distinguish between its own and other products' documentation.

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by floren on Wednesday June 05, 2024 @01:47PM (#40588031)
> As soon as I asked "are you giving me instructions for Gemini Advanced or Bard?" it was like "OH LOL WOOPS!! YOU GOT ME BRO! XD I CAN'T DO ANY OF THAT! LOL." Which, honestly, is great. Being able to evaluate its answers to realize it's wrong is really neat.

Did it evaluate its answers, or did your expression of doubt cause the eager-to-please language model to switch from "generate (wrong) instructions because that's what the user asked for" to "acknowledge an error because that's what the user asked for"?

How many times have we seen "Oops, you're right! 2 + 2 is actually 5! I apologize for saying it was 4 earlier!"

Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by chefandy on Wednesday June 05, 2024 @11:48PM (#40593250)
I suppose whether or not these things evaluate anything at all depends on your perspective, but as soon as I asked if it was giving me the correct information, it did actually give me useful information after that without further prompting.
Re:Alice in Wonderland: Simple Tasks Showing Complete Reason... by chefandy on Wednesday June 05, 2024 @12:25PM (#40586910)
> Yes this is a common thing I see people who think LLMs are idiots do.

If you're among technologists discussing LLMs academically, as we are, that's a reasonable approach. However, I see a lot of people fail to distinguish that from LLM-powerd products sold to the general public as intelligent bots that can understand your plain english and output answers.

People use their existing mental models when interacting with something. If you have 3 different interfaces with a widget to trigger the same exact function, but one look like a music play button, one looks like a gas pedal, and one looks like mechanical pinball plunger, we interact with those things differently because we know how those things work. In this context, chatbots are designed to engage people's existing mental model for chatting with a person via text. The further you stray from people's expectations of human chat, the further you are from people's expectations, for better, or worse.

If you're selling someone a product claiming it understands plain language questions and gives plain language answers, then not getting the right answer to that question makes it idiotic. The subtleties aren't within most users' grasp, and the "FYI: this thing might be full of shit" disclaimer isn't helpful if you don't know enough about what you're asking to administer a proper smell test.

Your statements are obviously not wrong, but I see people saying these things like its reasonable for non-technical end users to reason about those subtleties. Considering how those things are marketed, I really don't think it is.