The strangeness of this argument is its unnecessary for any one technique to be all in capable because a mixture of techniques is more powerful than one alone. A LLM forming an abductive “plan” and synthesizing results from agent functions that do the deductive reasoning, optimization, math, and other functions LLMs are bad at is remarkably powerful and is incredibly flexible in new semantic contexts. Who cares if LLMs can solve partial differential equations if they can establish a partial differential equation to solve given semantic context and a sufficient training set of semantically grounded PDEs ? That can be delegated to the last 70 years of research on numerical methods. Similarly we don’t expect a recursive decent parser to play chess, but you can absolutely use a parser to make a grammar that exposes a chess playing solver. The combination of the solver and the grammar that enables its use is more powerful than the sum of its parts.

LLMs neatly solve an enormous number of hitherto unsolved problems and yield untapped solutions to many more. Generative AI in general composes in powerful ways to unlock even more - semantic analysis of visual information expressible directly in structured language alone is powerful, as is exposing generative robotic motion and audio processing, paired with solvers, optimizers, etc - what’s the residual? Synthetic agency? We’ve had multi level goal based agents for a long time. One that’s reprogrammable by the generative model assembly?

(Personally, I am partial to the "alien chef" metaphor for AI: AIs are just reacting the way a chef from Earth would if transported to an alien planet where all the vegetables look the same (and ingredients) but are in reality very different from Earth, and have different effects, including poisonous. They think they are coming up with dishes fit for consumers, but the chef has no understanding of what's going on ...

A lot of the hype around LLMs and the unfounded promises of them solving complex tasks autonomously seem to stem from a lack of understanding on how these models actually work.

The tech is impressive and has its uses, but we shouldn’t pretend like a token predictor can somehow reason and plan.

I don't know how people can still be making this misguided comment. Predicting tokens well requires reasoning and planning.

"Brains are impressive and have their uses, but we shouldn't pretend like a muscle controller can somehow reason and plan."

> Predicting tokens well requires reasoning and planning.

Reason and planning regarding token prediction

It cannot reason about the context of its output. It only infers the most likely token to follow up.

Of course it can reason about the context of its output. I'm honestly not sure what you're trying to say.

Can you give me an example of a prompt that requires your definition of "reasoning"?

Have you ever tried to get whatever fancy LLM to write some code for you and had it generate code that was at the same time plausible to look at but complete bullshit?

This is what I am talking about. It can reason and plan for what is the likely token based off its training data. It is completely unable to evaluate the logic of the code it was generating. It can not reason on the context (in this case, programming).

The same is true for other domains. Law, Medicine, etc. the stricter the field, the less reliable LLMs are, because it cannot reason about the context of what it is writing.

That said, I like LLMs, and I think it was an interesting productivity tool, only having been hyped beyond any reasonable expectations. I find it more useful for less strict contexts (for example, creative writing).

I mean obviously predicting tokens in the context of LLM output requires planning. But planning tokens doesn’t generalize. This is evident when you train an LLM on a small data set and ask it about any unseen variation of it.

There are many examples of slightly modified popular riddles that are easy to solve by reasoning about them, but LLMs always fall back to the most likely output from their training data.

They absolutely can reason and plan; how do you suppose they predict the next token?

That they’re not autonomously solving complex tasks is a bit of a straw man though, and with a bit of creativity we can easily imagine them being combined with models and modalities that do provide executive function and autonomy.

This is one of those things I’d love to be wrong on. I think your point on combining them with other models is interesting.

Do you think that Markov chains can reason and plan? Dijkstra’s algorithm? Curious where you draw the line

Well, yes, reasoning and planning abilities exist on a spectrum, so it isn’t so much a matter of where to draw the line as a question of degree. As for LLMs, I think their reasoning and planning is some of the most powerful and human-like we’ve seen so far, even if the hidden mechanisms and constraints are different (in some cases, more limited, but in others, vastly superior).

Our brains however are highly modular (a “committee of idiots”) so who’s to say a portion, and even a significant one, doesn’t operate on similar principles?

For thought: Can DNA reason and plan?

Can a collection of around 1.5 billion interconnected cells that predictably respond to signals in their environment using simple rules? How about 86 billion? 36 trillion?

These are ballpark counts of cells in crow’s brain, a human’s brain, and a human body. The question is, is it the cells themselves doing the reasoning and planning, or are they just the machinery this disembodied process happens to be running on? I’d argue intelligence is a distributed phenomena that our DNA is as much a party to as our brains.

Certainly the question of whether humans use DNA to reproduce or DNA uses humans is a matter of perspective.

There's no reason they can't. In fact LLMs do seem to be the best at artificial general-purpose reasoning of any attempt so far (pretty much every other attempt has failed miserably). That doesn't mean they're particularly good compared to humans, though. I'd characterize them as very high knowledge, low intelligence.

Sure the current generation, but what happens if we expand its scope, so instead of predicting the next token one at a time, it was allowed greater scope? What happens when you feed it summaries of its work as training data so it has long a short term memory, what happens when you feed several networks together creating a dialogue similar to the theories of a bicameral mind, what if we instead of having the llm halt after every prompt it was put in a loop? Its own output along with outside information used as the prompts for fallowing loop