News for hackers. Stuff that matters.
Re:The "Chain of Thought" Delusions by FeepingCreature on Sunday May 12, 2024 @06:39AM (#40333398)
Far as I can tell, all this demonstrates is there's tasks that CoT doesn't work for (yet)? If that? Or just that there's bad prompts? Really, the Twitter replies have already made these points so there's nothing to add. This just seems like research that's been written conclusion-first.

> Ever since I came across the Chain of Though (CoT) for LLMs paper, I wondered how it can possibly make sense given that there is little reason to believe that LLMs can follow procedures and unroll them for the current problem at hand. (After all, if they can do that, they should be able to do verification and general reasoning too--and we know they suck at those).

I don't follow. Why would procedure following imply general reasoning?

Re:The "Chain of Thought" Delusions by YeGoblynQueenne on Sunday May 12, 2024 @07:08AM (#40333470)
>> Far as I can tell, all this demonstrates is there's tasks that CoT doesn't work for (yet)?

What he says is that they can solve blocks world problems if you take them by the hand and show them how to do it, for every single problem you wan them to solve. Which is not very useful at all. He's arguing that this is what CoT achieves: it takes the LLM by the hand and introduces domain knowledge that the user has in every step of the way, so that it's not the LLM that's solving anything but the person using the LLM with CoT.

Do read the linked paper, it goes over this in more detail. It's a bit unfortunate that Rao chooses to communicate through the very noisy medium of twitter, but that's the internet.

Re:The "Chain of Thought" Delusions by FeepingCreature on Sunday May 12, 2024 @07:16AM (#40333496)
I think LLMs and CoT are powerful, but I agree with this description of what they do.

The important step is that they demonstrate that there are no architectural limits that keep the LLM from acting in these domains, "only" knowledge/planning ones. Once we have a big enough dataset of CoT prompts, the model "just" has to generalize CoT prompting, not CoT following. It decomposes the problem into two halves, and demonstrates that if one half (instruction generation) is provided, the other (instruction following) becomes very tractable.

Re:The "Chain of Thought" Delusions by YeGoblynQueenne on Sunday May 12, 2024 @07:31AM (#40333556)
Thanks for clarifying. That's not the claim that Rao is tweeting about though. Rao as far as I can tell is responding to a stream of papers that appeared in the last couple of years claiming that LLMs can plan right now, whether using CoT, or not. See the paper I linked to my reply to amenhotep (!) below for an example.

As to blocks world planning in particular, LLMs already have plenty of examples of block stacking problems - those are the standard motivational experiment in planning papers, like solving mazes is for Reinforcement Learning. Google returns 10 pages of results for "blocks world planning". If LLMs were capable of generalising as well as you expect they will one day, they should already be capable of solving block stacking problems without CoT and with no guidance.

Re:The "Chain of Thought" Delusions by FeepingCreature on Sunday May 12, 2024 @07:48AM (#40333646)
Saying that LLMs should be able to do problems without CoT is a bit like saying that humans should be able to write programs without thinking though. CoT is fundamentally an architectural necessity owed to the finite cognitive effort possible per token. Let it be CoT, let it be QuietSTaR ( https://arxiv.org/abs/2403.09629 ), let it be pause tokens ( https://arxiv.org/abs/2310.02226 ), let it even be dots ( https://arxiv.org/abs/2404.15758 ), but the thinking has to happen somewhere.

I think mostly, LLMs at the moment are incredibly uneven. LLM assistants can pull obscure knowledge out of nowhere one second and fail extremely basic reasoning the next. So just because some example happens a lot in the source material doesn't mean the LLM can learn it. IMO, that CoT works at all is more down to luck of the training set than any inherent capability of the LLM.

Re:The "Chain of Thought" Delusions by amenhotep on Sunday May 12, 2024 @06:52AM (#40333434)
It struck me as really weird that he seemed to assume implicitly that his Blocksworld problems would be something an LLM should be able to do and presented the failures as shocking; one look at them immediately gives me the impression that they are the kind of problem LLMs tend to be pathologically bad at.

Probably he's better tuned in to the tone of the discourse than me but I feel like if you showed CoT helped with Blocksworld then that would be a huge and triumphant result, rather than something taken for granted that we must loudly disprove!

Re:The "Chain of Thought" Delusions by YeGoblynQueenne on Sunday May 12, 2024 @07:13AM (#40333486)
From the tweet:

>> And yet the practitioners of CoT swear that any and every problem can be solved with LLM by giving it a bit of a CoT help.

For example, see this arxiv paper:

Generalized Planning in PDDL Domains with Pretrained Large Language Models

https://arxiv.org/abs/2305.11014

Where the authors conclude:

In this work, we showed that GPT-4 with CoT summarization and automated debugging is a surprisingly strong generalized planner in PDDL domains.

The author of the tweet is an expert on planning and he's responding to that kind of thing.

Re:The "Chain of Thought" Delusions by FeepingCreature on Sunday May 12, 2024 @07:54AM (#40333689)
I think these scenarios are compatible if we view LLMs as "fragile reasoners": they can occasionally reason, but it is an intermittent state that is easily disturbed. In such a world, we would expect to see that people who want LLMs to work can make them work with difficulty, and people who want or expect LLMs to fail can make them fail easily - or rather, maybe less adversarially phrased, one can generate examples of either outcome.
Re:The "Chain of Thought" Delusions by mistermann on Sunday May 12, 2024 @11:07AM (#40335085)
>> And yet the practitioners of CoT swear that any and every problem...

> Where the authors conclude:

> ...is a surprisingly strong generalized planner

Wtf is going on here?