News for hackers. Stuff that matters.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by rjbwork on Wednesday October 30, 2024 @07:53PM (#42001927)
>For the purpose of AGI, LLM are starting to look like a local maximum.

I've been saying it since they started popping off last year and everyone was getting euphoric about them. I'm basically a layman - a pretty good programmer and software engineer, and took a statistics and AI class 13 years ago in university. That said, it just seems so extremely obvious to me that these things are likely not the way to AGI. They're not reasoning systems. They don't work with axioms. They don't model reality. They don't really do anything. They just generate stochastic output from the probabilities of symbols appearing in a particular order in a given corpus.

It continues to astound me how much money is being dumped into these things.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by ChadNauseam on Wednesday October 30, 2024 @08:00PM (#42001981)
How do you know that they don’t do these things? Seems hard to say for sure since it’s hard to explain in human terms what a neural network is doing.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by nephy on Wednesday October 30, 2024 @08:04PM (#42002031)
If you give an LLM a word problem that involves the same math and change the names of the people in the word problem the LLM will likely generate different mathematical results. Without any knowledge of how any of this works, that seems pretty damning of the fact that LLMs do not reason. They are predictive text models. That’s it.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by Workaccount2 on Wednesday October 30, 2024 @08:21PM (#42002137)
This is a relatively trivial task for current top models.

More challenging are unconventional story structures, like a mom named Matthew with a son named Mary and a daughter named William, who is Matthew's daughter?

But even these can still be done by the best models. And it is very unlikely there is much if any training data that's like this.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by alexwebb2 on Wednesday October 30, 2024 @08:38PM (#42002235)
That's a neat example problem, thanks for sharing!

For anyone curious: https://chatgpt.com/share/6722d130-8ce4-800d-bf7e-c1891dfdf7...

> Based on traditional naming conventions, it seems that the names might have been switched in this scenario. However, based purely on your setup:

>

> Matthew has a daughter named William and a son named Mary.

>

> So, Matthew's daughter is William.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by rileymat2 on Wednesday October 30, 2024 @11:15PM (#42003142)
How do people fair on unconventional structures? I am reminded of that old riddle involving a the mother being the doctor after a car crash.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by adwn on Thursday October 31, 2024 @03:31AM (#42004270)
No idea why you've been downvoted, because that's a relevant and true comment. A more complex example would be the Monty Hall problem [1], for which even some very intelligent people will intuitively give the wrong answer, whereas symbolic reasoning (or Monte Carlo simulations) leads to the right conclusion.

[1] https://en.wikipedia.org/wiki/Monty_Hall_problem

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by alexwebb2 on Wednesday October 30, 2024 @08:10PM (#42002065)
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by TZubiri on Wednesday October 30, 2024 @08:53PM (#42002317)
It's worth noting that this may not be result of a pure LLM, it's possible that ChatGPT is using "actions", explicitly:

1- running the query through a classifier to figure out if the question involves numbers or math 2- Extract the function and the operands 3- Do the math operation with standard non-LLM mechanisms 4- feed back the solution to the LLM 5- Concatenate the math answer with the LLM answer with string substitution.

So in a strict sense this is not very representative of the logical capabilities of an LLM.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by thomashop on Thursday October 31, 2024 @12:04AM (#42003389)
It shows you when it's calling functions. I also did the same test with Llama, which runs locally and cannot access function calls and it works.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by TZubiri on Thursday October 31, 2024 @02:27AM (#42004011)
You are right I actually downloaded Llama to do more detailed tests. God bless Stallman.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by digging on Thursday October 31, 2024 @11:06AM (#42007670)
Then what's the point of ever talking about LLM capabilities again? We've already hooked them up to other tools.

This confusion was introduced at the top of the thread. If the argument is "LLMs plus tooling can't do X," the argument is wrong. If the argument is "LLMs alone can't do X," the argument is worthless. In fact, if the argument is that binary at all, it's a bad argument and we should laugh it out of the room; the idea that a lay person uninvolved with LLM research or development could make such an assertion is absurd.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by TaylorAlexander on Wednesday October 30, 2024 @10:01PM (#42002702)
At this point I really only take rigorous research papers in to account when considering this stuff. Apple published research just this month that the parent post is referring to. A systematic study is far more compelling than an anecdote.

https://machinelearning.apple.com/research/gsm-symbolic

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by og_kalu on Wednesday October 30, 2024 @10:11PM (#42002754)
That study shows 4o, o1-mini and o1-preview's new scores are all within margin error on 4/5 of their new benchmarks(some even see increases). The one that isn't involves changing more than names.

Changing names does not affect the performance of Sota models.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by gruez on Wednesday October 30, 2024 @10:18PM (#42002797)
>That study very clearly shows 4o, o1-mini and o1-preview's new scores are all within margin error on 4/5 of their new benchmarks.

Which figure are you referring to? For instance figure 8a shows a -32.0% accuracy drop when an insignificant change was added to the question. It's unclear how that's "within the margin of error" or "Changing names does not affect the performance of Sota models".

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by og_kalu on Wednesday October 30, 2024 @10:29PM (#42002850)
Table 1 in the Appendix. GSM-No-op is the one benchmark that sees significant drops for those 4 models as well (with preview dropping the least at -17%). No-op adds "seemingly relevant but ultimately inconsequential statements". So "change names, performance drops" is decidedly false for today's state of the art.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by gruez on Wednesday October 30, 2024 @10:39PM (#42002919)
Thanks. I wrongly focused on the headline result of the paper rather than the specific claim in the comment chain about "changing name, different results".
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by TaylorAlexander on Thursday October 31, 2024 @03:24AM (#42004236)
Ah, that’s a good point thanks for the correction.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by zmgsabst on Thursday October 31, 2024 @10:47AM (#42007471)
Only if there isn’t a systemic fault, eg bad prompting.

Their errors appear to disappear when you correctly set the context from conversational to adversarial testing — and Apple is actually testing the social context and not its ability to reason.

I’m just waiting for Apple to release their GSM-NoOp dataset to validate that; preliminary testing shows it’s the case, but we’d prefer to use the same dataset so it’s an apples-to-apples comparison. (They claim it will be released “soon”.)

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by gruez on Wednesday October 30, 2024 @10:12PM (#42002758)
To be fair, the claim wasn't that it always produced the wrong answer, just that there exists circumstances where it does. A pair of examples where it was correct hardly justifies a "demonstrably false" response.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by thomashop on Thursday October 31, 2024 @03:30AM (#42004265)
Conversely, a pair of examples where it was incorrect hardly justifies the opposite response.

If you want a more scientific answer there is this recent paper: https://machinelearning.apple.com/research/gsm-symbolic

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by EraYaN on Thursday October 31, 2024 @06:28AM (#42005336)
It kind of does though, because it means you can never trust the output to be correct. The error is a much bigger deal than it being correct in a specific case.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by thomashop on Thursday October 31, 2024 @10:10AM (#42007097)
You can never trust the outputs of humans to be correct but we find ways of verifying and correcting mistakes. The same extra layer is needed for LLMs.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by digging on Thursday October 31, 2024 @11:09AM (#42007691)
> It kind of does though, because it means you can never trust the output to be correct.

Maybe some HN commenters will finally learn the value of uncertainty then.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by astrange on Thursday October 31, 2024 @04:25AM (#42004561)
Minor edits to well known problems do easily fool current models though. Here's one 4o and o1-mini fail on, but o1-preview passes. (It's the mother/surgeon riddle so kinda gore-y.)

https://chatgpt.com/share/6723477e-6e38-8000-8b7e-73a3abb652...

https://chatgpt.com/share/6723478c-1e08-8000-adda-3a378029b4...

https://chatgpt.com/share/67234772-0ebc-8000-a54a-b597be3a1f...

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by _flux on Thursday October 31, 2024 @04:55AM (#42004711)
I think you didn't use the "share" function; I cannot open any of these links. Can you do it in a private browser session (so you're not logged in)?
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by astrange on Thursday October 31, 2024 @05:03AM (#42004754)
Oops, fixed the links.

mini's answer is correct, but then it forgets that fathers are male in the next sentence.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by jklinger410 on Wednesday October 30, 2024 @08:28PM (#42002177)
This is what kind of comments you make when your experience with LLMs is through memes.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by vanviegen on Thursday October 31, 2024 @03:34AM (#42004288)
And yet, humans, our benchmark for AGI, suffer from similar problems, with our reasoning being heavily influenced by things that should have been unrelated.

https://en.m.wikipedia.org/wiki/Priming_(psychology)

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by FuckButtons on Wednesday October 30, 2024 @09:44PM (#42002584)
Absence of evidence or a simple explanation does not mean that you can imbue statistical regression with animal spirits.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by toasterlovin on Wednesday October 30, 2024 @10:46PM (#42002961)
The burden of proof goes both ways: if you want to say X isn’t really the same thing as human general intelligence, you have to be able to confidently say human general intelligence isn’t really the same thing as X.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by beardedwizard on Thursday October 31, 2024 @09:02AM (#42006497)
An interesting mental trap, except that the indirect evidence keeps mounting that LLMs do not possess human general intelligence, even if we can not describe exactly how it exists in the brain.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by toasterlovin on Thursday October 31, 2024 @12:30PM (#42008477)
On the contrary, the parallels between the peculiarities of LLMs and various aspects of human cognition seem very striking to me. Given how early we are in figuring out what we can accomplish with LLMs, IMO the appropriate epistemic stance is to not reach any unequivocal conclusions. And then my personal hunch is that LLMs may be most of the magic, with how they're orchestrated and manipulated being the remainder (which may take a very long time to figure out).
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by TZubiri on Friday November 01, 2024 @04:17PM (#42021111)
I think it's just that I understand LLMs better than you, and I know that they are very different from human intelligence. Here's a couple of differences:

- LLMs use fixed resources when computing an answer. And to the extent that they don't, they are function calling and the behaviour is not attributable to the LLMs. For example when using a calculator, it is displaying calculator intelligence.

- LLMs do not have memory, and if they do it's very recent and limited, and distinct from any being so far. They don't remember what you said 4 weeks ago, and they don't incorporate that into their future behaviour. And if they do, the way they train and remember is very distinct from that of humans and relates to it being a system being offered as a free service to multiple users. Again to the extent that they are capable of remembering, their properties are not that of LLMs and are attributable to another layer called via function calling.

LLMs are a perception layer for language, and perhaps for output generation, but they are not the intelligence.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by broast on Thursday October 31, 2024 @08:34AM (#42006243)
Are you not imbueing animals with spirits based on lack of evidence of statistical regression?
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by _heimdall on Wednesday October 30, 2024 @10:50PM (#42002985)
The whole design of an LLM is to consume and compress a huge space of human-generared content and use that to predict how a human would reply, one token at a time. That alone means the LLM isn't modelling anything beyond the human content it was trained on, and there is no reasoning since every prediction is based only on probabilities combined with controls similar to randomization factors used to avoid an entirely deterministic algorithm.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by ChadNauseam on Wednesday October 30, 2024 @11:43PM (#42003282)
For a lot of the content they were trained on, it seems like the easiest way to predict the next token would be to model the world or work with axioms. So how do we know that an LLM isn't doing these things internally?
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by thomashop on Thursday October 31, 2024 @12:00AM (#42003370)
In fact, it looks like the model is doing those things internally.

  We argue that representations in AI models, particularly deep networks, are converging. First, we survey many examples of convergence in the literature: over time and across multiple domains, the ways by which different neural networks represent data are becoming more aligned. Next, we demonstrate convergence across data modalities: as vision models and language models get larger, they measure distance between datapoints in a more and more alike way. We hypothesize that this convergence is driving toward a shared statistical model of reality, akin to Plato’s concept of an ideal reality. We term such a representation the platonic representation and discuss several possible selective pressures toward it. Finally, we discuss the implications of these trends, their limitations, and counterexamples to our analysis.
https://arxiv.org/html/2405.07987v5
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by _heimdall on Thursday October 31, 2024 @09:30AM (#42006715)
Unless I misread this paper, their argument is entirely hypothetical. Meaning that the LLM is still a black box and they can only hypothesise what is going internally by viewing the output(s) and guessing at what it would take to get there.

There's nothing wrong with a hypothesis or that process, but it means we still don't know whether models are doing this or not.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by thomashop on Thursday October 31, 2024 @10:30AM (#42007285)
Maybe I mixed up that paper with another but the one I meant to post shows that you can read something like a world model from the activations of the layers.

There was a paper that shows a model trained on Othello moves creates a model of the board, models the skill level of their opponent and more.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by _heimdall on Thursday October 31, 2024 @08:15AM (#42006071)
Well my understanding is that there's ultimately the black box problem. We keep building these models and the output seems to get better, but we can't actually inspect how they work internally.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by ricardobeat on Wednesday October 30, 2024 @11:20PM (#42003166)
That’s not an accurate description. Attention / multi-head attention mechanisms allow the model to understand relationships between words far apart and their context.

They still lack, as far as we know, a world model, but the results are already eerily similar to how most humans seem to think - a lot of our own behaviour can be described as “predict how another human would reply”.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by thomashop on Thursday October 31, 2024 @03:27AM (#42004251)
When trained on simple logs of Othello's moves, the model learns an internal representation of the board and its pieces. It also models the strength of its opponent.

https://arxiv.org/abs/2210.13382

I'd be more surprised if LLMs trained on human conversations don't create any world models. Having a world model simply allows the LLM to become better at sequence prediction. No magic needed.

There was another recent paper that shows that a language model is modelling things like age, gender, etc., of their conversation partner without having been explicitly trained for it

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by _heimdall on Thursday October 31, 2024 @08:14AM (#42006056)
Do we know for a fact that the mechanisms are actually used that way inside the model?

My understand was that they know how the model was designed to be able to work, but that there's been very little (no?) progress in the black box problem so we really don't know much at all about what actually happens internally.

Without better understanding of what actually happens when an LLM generates an answer I stick with the most basic answer that its simply predicting what a human would say. I could be wildly misinformed there, I don't work directly in the space and its been moving faster than I'm interested in keeping up with.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by wg0 on Thursday October 31, 2024 @09:27AM (#42006688)
How do we know Santa doesn't exist? Maybe he does.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by alexwebb2 on Wednesday October 30, 2024 @08:01PM (#42001996)
If you expect "the right way" to be something _other_ than a system which can generate a reasonable "state + 1" from a "state" - then what exactly do you imagine that entails?

That's how we think. We think sequentially. As I'm writing this, I'm deciding the next few words to type based on my last few.

Blows my mind that people don't see the parallels to human thought. Our thoughts don't arrive fully formed as a god-given answer. We're constantly deciding the next thing to think, the next word to say, the next thing to focus on. Yes, it's statistical. Yes, it's based on our existing neural weights. Why are you so much more dismissive of that when it's in silicon?

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by Techonomicon on Wednesday October 30, 2024 @08:29PM (#42002188)
Because we still don't know how the brain really does all it does in very specific terms, so why assume to know exactly how we think?
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by alexwebb2 on Wednesday October 30, 2024 @08:34PM (#42002209)
Why is there only one valid way of producing thoughts?
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by jltsiren on Wednesday October 30, 2024 @09:59PM (#42002687)
Finite-state machines are a limited model. In principle, you can use them to model everything that can fit in the observable universe. But that doesn't mean they are a good model for most purposes.

The biggest limitation with the current LLMs is the artificial separation between training and inference. Once deployed, they are eternally stuck in the same moment, always reacting but incapable of learning. At best, they are snapshots of a general intelligence.

I also have a vague feeling that a fixed set of tokens is a performance hack that ultimately limits the generality of LLMs. That hardcoded assumptions make tasks that build on those assumptions easier and seeing past the assumptions harder.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by alexwebb2 on Wednesday October 30, 2024 @10:37PM (#42002905)
> At best, they are snapshots of a general intelligence.

So are we, at any given moment.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by Jensson on Wednesday October 30, 2024 @11:03PM (#42003068)
> As I'm writing this, I'm deciding the next few words to type based on my last few.

If so you could have written this as a newborn baby, you are determining these words based on a lifetime of experience. LLMs doesn't do that, every instance of ChatGPT is the same newborn baby while a thousand clones of you could all be vastly different.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by thomashop on Thursday October 31, 2024 @12:06AM (#42003403)

  We argue that representations in AI models, particularly deep networks, are converging. First, we survey many examples of convergence in the literature: over time and across multiple domains, the ways by which different neural networks represent data are becoming more aligned. Next, we demonstrate convergence across data modalities: as vision models and language models get larger, they measure distance between datapoints in a more and more alike way. We hypothesize that this convergence is driving toward a shared statistical model of reality, akin to Plato’s concept of an ideal reality. We term such a representation the platonic representation and discuss several possible selective pressures toward it. Finally, we discuss the implications of these trends, their limitations, and counterexamples to our analysis.
https://arxiv.org/html/2405.07987v5
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by csomar on Wednesday October 30, 2024 @09:54PM (#42002653)
I am pretty sure Claude 3.5 Sonnet can reason or did reason with a particular snippet of code I was working on. I am not an expert in this area but my guessing is that these neural nets (made for language prediction) are being used for reasoning. But that’s not their optimal behavior (since they are token predictor). A big jump in reasoning will happen when reasoning is off loaded to an LRM.

Human brains are sure big but they are inefficient because a big portion of the brain is going to non-intelligence stuff like running the body internal organs, eye vision, etc…

I do agree that the money is not well spent. They should haver recognized that we are hitting s local maximum with the current model and funding should be going to academic/theoretical instead of dump brute force.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by wyldfire on Wednesday October 30, 2024 @10:04PM (#42002718)
> It continues to astound me how much money is being dumped into these things.

Maybe in our society there's a surprising amount of value of a "word stirrer" intelligence. Sure, if it was confident when it was right and hesitant when it was wrong it'd be much better. Maybe humans are confidently wrong often enough that an artificial version that's compendious experience to draw on is groundbreaking.

Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by chamomeal on Wednesday October 30, 2024 @11:37PM (#42003247)
I totally agree that they’re a local maximum and they don’t seem like a path to AGI. But they’re definitely kinda reasoning systems, in the sense that they can somewhat reason about things. The whacky process they use to get there doesn’t take away from that IMO
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by kibwen on Thursday October 31, 2024 @09:55AM (#42006946)
> I've been saying it since they started popping off last year and everyone was getting euphoric about them.

Remember the resounding euphoria at the LK-99 paper last year, and how everyone suddenly became an expert on superconductors? It's clear that we've collectively learned nothing from that fiasco.

The idea of progress itself has turned into a religious cult, and what's worse, "progress" here is defined to mean "whatever we read about in 1950s science fiction".