I found the following transcript of the thought process to that question equally enlightening and funny:

Thought for 4 seconds

1 problem understanding

The task is to count the number of instances of the letter 'r' in the word 'strawerberry'. This requires identifying each occurrence of 'r' in the given word and keeping a count.

2 data gathering

The word to analyze is 'strawerberry'. I need to examine each character in this word.

3 hypothesis formation

Based on a quick glance, I hypothesize that there are 2 instances of 'r' in 'strawerberry'.

4 analysis

Let's break down the word into individual characters: s, t, r, a, w, e, r, b, e, r, r, y

5 calculation

Now, let's count the 'r's: 1) First 'r' at position 3, 2) Second 'r' at position 7, 3) Third 'r' at position 10. Total count: 3

6 verification

Double-checking the count: s(0), t(0), r(1), a(0), w(0), e(0), r(2), b(0), e(0), r(3), r(3), y(0). Confirmed: 3 instances of 'r'.

7 error checking

The initial hypothesis was incorrect. The actual count (3) differs from the initial guess (2). This highlights the importance of careful analysis rather than quick assumptions.

8 conclusion drawing

The letter 'r' appears 3 times in the word 'strawerberry'.

9 implementation

The final response should be the number 3, as this is the correct count of 'r' instances in 'strawerberry'.

I get:

>Alright, I need to figure out how many 'r's are in the word 'strawerberry'. Let's look at it carefully. I'll write it down: s-t-r-a-w-e-r-b-e-r-r-y. Now, I'll go through each letter one by one.

>Starting from the beginning: s, t, r – that's one 'r'. Then a, w, e, r – that's the second 'r'. Next, b, e, r – that's the third 'r'. And finally, another r before the y, making it the fourth 'r'. So, I count four 'r's in total.

>The word "strawberry" contains four 'r's.

Wow, so close. I wonder what went wrong here?

Tokenization, still. LLMs don’t see individual letters. It’s like using Google translate to ask a German speaker the same question and concluding that Germans can’t count letters.

Does this mean there's no actual reasoning happening in these "reasoning" steps?

Well, the model is based on llama-8b, which is quite bad at reasoning. Reasoning (or things that look and quack like reasoning) is more the domain of 70B+ models, and some of the newer 7B models.

The model is doing well on many reasoning tasks, what they are doing is a massive step up from llama-8b. But it still makes some silly mistakes. I bet if you did the same finetuning procedure with quen-7B or llama-70B as a starting point you would get a quite competent model

Of course there’s no actual reasoning.

Is there something more than prediction going on? I'd like to understand why there would be studies like this: https://arxiv.org/pdf/2307.02477 if it's all just prediction. Are there some algorithms that are called "reasoning", that are somewhere between prediction and full-blown reasoning?

My guess is the framing in that paper was chosen to be provocative. The authors surely understand deep neural networks neither reason nor recite as we understand it in humans.

In any case, the conclusion was decidedly against reasoning.