> Yes, these responses are annoying, but what's your point?

The point I imagine is that there is no reasoning going on at all. Some humans sometimes struggle with some reasoning, of course. That is completely irrelevant to whether LLMs reason.

Picking word sequences that are most likely acceptable based on a static model formed months ago is not reasoning. No model is being constructed on the fly, no patterns recognised and extrapolated.

There are useful things possible of course but these models will never offer more than a nice user interface to a static model. They don't reason.

And so, what do you think reasoning is? Or how would you know if something can reason or not ?

Why do you say that isn't reasoning, and what do you think human reasoning is?

I do think you have a point that the lack of a working memory is a severe constraints, but I also think you are wrong that these models will remain a user interface to a static model rather than being given the ability to add working memory and form long term memories and reason with that.

I also think it's an entirely open question whether they are reasoning under a reasonable definition, in part because we don't have one, and I think any claim that they don't reason ironically comes from a lack of reasoning about the high degree of uncertainty and ambiguity we have with respect to what reasoning means and how to measure it.

> Why do you say that isn't reasoning, and what do you think human reasoning is?

One worthwhile definition would be the ability to recognise patterns in knowledge and apply them to new context to generate new knowledge. There is none of this kind of processing happening despite how believable some of the words sometimes are.

To me, under this definitions LLMs are then clearly and obviously reasoning based on many conversations I've had.

E.g. the ability to solve a problem in code and then translate it to a new made up programming language described to it would easily qualify to me.

And this is a task a whole lot of humans would be unable to carry out.

If they were actually reasoning, tests like the GP's would show it. They don't connect dots, they can be prompted to select different pathways through their static model and that selection can be based on a pretty small context but nothing about that model changes. Tomorrow's conversation is only different based on rand(). LLMs have a very large static model and confusing that with reasoning is fairly common but still incorrect.

This is not valid logic. If they are reasoning, tests like GPs might show it. Failing the test, however, can have many other causes: They can just not be good enough at reasoning, or they might be failing because they see tokens and have had too little training to connect that to both the spelling in sounds in a way that generalize.

I'd be willing to bet a whole lot of humans would fail that test too, because a lot of people are really bad at applying a rule without practicing on examples first, and so often struggle to take feedback without examples. If they did, would you claim they can't reason?

Your claim to know that LLMs are not is not based in fact, but speculation that too me is itself not based in reasoning. Should I question your ability to reason because I don't think you've done so in this argument?

We can inspect these models and look at exactly what they're doing. We don't need to guess about why they fail tests like this. We know as a fact that they are static and not generating new knowledge from input.

I don't understand why you keep talking about human capabilities - your guesses about what humans may or may not be able to do are irrelevant. You can hold whatever opinion you like about my ability to reason, but I'd suggest using less wishful thinking with regards LLMs.

They're very useful, but not for reasoning.

We can inspect these models, and repeated attempts at that shows that they e.g. have built internal models from learning that generalise, so I'm not sure what you think that tells you that justifies arguing they can't reason. Beyond that, trying to reverse the specific "reasoning" that leads to a given output is generally hard, and certainly no attempts at doing so have produced any evidence that the way they work is conclusively not reasoning as far as I am aware.

> We know as a fact that they are static and not generating new knowledge from input.

We know the models in themselves if not wired up to be finetuned during operation are static. This is not a property of an LLM but of the environment they run in. We know the second is wrong - they produce output that often contain new knowledge. That this output needs to be fed back in as context to act as short term memory in common setups like ChatGPT where finetuning does not happen automatically during operation does not mean it is not produced.

> I don't understand why you keep talking about human capabilities - your guesses about what humans may or may not be able to do are irrelevant. You can hold whatever opinion you like about my ability to reason, but I'd suggest using less wishful thinking with regards LLMs.

I keep talking about human capabilities because I presume that you would argue that there are humans of normal intellect incapable of reasoning.

To be able to assert with any confidence that LLMs do not reason you need a definition of reasoning that LLMs can not (not just do not in a single test) meet, but that won't result in claiming there are a lot of people around who can't reason.

Am I wrong? Do you believe there are humans that do not clear the bar and are unable to reason?

To me, a "chat-style" setup of an LLM that provides a feedback loop and memory through context, albeit a small one, clears the bar you set for reasoning with ease and is able to extrapolate and reason about e.g. software at a level that sometimes - but certainly not always, nor consistently - exceeds what I see from experienced developers.

That it also fails does not alter that part - humans fail to apply reasoning all the time. Depressingly often, if anything.

> You can hold whatever opinion you like about my ability to reason, but I'd suggest using less wishful thinking with regards LLMs.

Nothing I've said here is wishful thinking. All I've done is point to direct experience combined with pointing out that there is no evidence for the claim that they are not able to reason, and that the arguments set forward for that claim here have not been logically sound.

I will say that in my opinion they can reason by my subjective idea of what reasoning means, without necessarily being able to precisely define that, but I also will not argue it objectively true that they can reason as that is equally problematic without first defining reasoning in an objectively measurable way (needed, because as you can tell, we disagree on whether they clear your bar - to me your bar the way you described it is trivial for them to meet)

To me, a whole lot of the discussions in this thread are evidence of how exceedingly low the bar for what is reason needs to be for us not to have to exclude a whole lot of people as unable to reason. Humans get hung up on ideas and refuse to budge all the time - I do it too, all the time - and refuse to take in new information as a result, and fail to generalise, and keep making flawed arguments as a result all the time. Yet we would generally not claim that this means people are unable to reason even when it gets to the level where we might think that a person does not reason in that specific case.

That in itself does not mean they can reason, but to me the typical arguments claiming they can't reason tends to be exceedingly poorly reasoned.

Pointing out the static nature of the models gets closer, and is perhaps the best argument against their reasoning ability I've heard, but is weak both because chat-style models effectively use context as short-term memory and so you need to assess model+context, and because it's not a qualitative limitation of the model architecture but of the sandbox we've put it in where we don't continue fine-tuning from the conversations in real-time. Yet even so, there have been humans without ability to form long-term memory, and I doubt you'd argue they were unable to reason.

> They're very useful, but not for reasoning.

To me, they have been very useful for their reasoning ability in a long range of cases. It's hit and miss. LLMs are extremely dumb in some areas, and do well in others. Using them blindly and just assuming they'll do well in a given test will not work. Hence the point that it is not logically sound to argue that they are unable to reason because they failed to generalise in a specific test, because if so we would then need to conclude that most humans (myself included) can't reason because we all fail to do so on a regular basis.

My example of extrapolating from a simple description of a (non-existing) programming language to being able to translate programs into it or explain how one works and reason about the design tradeoffs, or even symbolically "execute" it and tell me what the output would be, for example, is one I know from first han experience using it as a means to assess analytical capabilities of even quite experienced developers is something a lot of really smart people struggle with, but where when I've experimented with GPT4 have gotten good results.