I don't know how people can still be making this misguided comment. Predicting tokens well requires reasoning and planning.

"Brains are impressive and have their uses, but we shouldn't pretend like a muscle controller can somehow reason and plan."

> Predicting tokens well requires reasoning and planning.

Reason and planning regarding token prediction

It cannot reason about the context of its output. It only infers the most likely token to follow up.

Of course it can reason about the context of its output. I'm honestly not sure what you're trying to say.

Can you give me an example of a prompt that requires your definition of "reasoning"?

Have you ever tried to get whatever fancy LLM to write some code for you and had it generate code that was at the same time plausible to look at but complete bullshit?

This is what I am talking about. It can reason and plan for what is the likely token based off its training data. It is completely unable to evaluate the logic of the code it was generating. It can not reason on the context (in this case, programming).

The same is true for other domains. Law, Medicine, etc. the stricter the field, the less reliable LLMs are, because it cannot reason about the context of what it is writing.

That said, I like LLMs, and I think it was an interesting productivity tool, only having been hyped beyond any reasonable expectations. I find it more useful for less strict contexts (for example, creative writing).

I mean obviously predicting tokens in the context of LLM output requires planning. But planning tokens doesn’t generalize. This is evident when you train an LLM on a small data set and ask it about any unseen variation of it.

There are many examples of slightly modified popular riddles that are easy to solve by reasoning about them, but LLMs always fall back to the most likely output from their training data.

They absolutely can reason and plan; how do you suppose they predict the next token?

That they’re not autonomously solving complex tasks is a bit of a straw man though, and with a bit of creativity we can easily imagine them being combined with models and modalities that do provide executive function and autonomy.

This is one of those things I’d love to be wrong on. I think your point on combining them with other models is interesting.

Do you think that Markov chains can reason and plan? Dijkstra’s algorithm? Curious where you draw the line

Well, yes, reasoning and planning abilities exist on a spectrum, so it isn’t so much a matter of where to draw the line as a question of degree. As for LLMs, I think their reasoning and planning is some of the most powerful and human-like we’ve seen so far, even if the hidden mechanisms and constraints are different (in some cases, more limited, but in others, vastly superior).

Our brains however are highly modular (a “committee of idiots”) so who’s to say a portion, and even a significant one, doesn’t operate on similar principles?

For thought: Can DNA reason and plan?

Can a collection of around 1.5 billion interconnected cells that predictably respond to signals in their environment using simple rules? How about 86 billion? 36 trillion?

These are ballpark counts of cells in crow’s brain, a human’s brain, and a human body. The question is, is it the cells themselves doing the reasoning and planning, or are they just the machinery this disembodied process happens to be running on? I’d argue intelligence is a distributed phenomena that our DNA is as much a party to as our brains.

Certainly the question of whether humans use DNA to reproduce or DNA uses humans is a matter of perspective.

There's no reason they can't. In fact LLMs do seem to be the best at artificial general-purpose reasoning of any attempt so far (pretty much every other attempt has failed miserably). That doesn't mean they're particularly good compared to humans, though. I'd characterize them as very high knowledge, low intelligence.

Sure the current generation, but what happens if we expand its scope, so instead of predicting the next token one at a time, it was allowed greater scope? What happens when you feed it summaries of its work as training data so it has long a short term memory, what happens when you feed several networks together creating a dialogue similar to the theories of a bicameral mind, what if we instead of having the llm halt after every prompt it was put in a loop? Its own output along with outside information used as the prompts for fallowing loop