It does more than that. It understands how to do basic math. You can ask it what ((935+91218)/4)*3) is and it will answer it correctly. Swap those numbers for any other random numbers, it will answer it correctly.
It has never seen that during training, but it understands the mathematical concepts.
If you ask ChatGPT how it does this, it says "I break down the problem into its component parts, apply relevant mathematical rules and formulas, and then generate a solution".
It's that "apply mathetmatical rules" part that is more than just, essentially, filling in the next likely token.
At least for GPT-3, during my own experimentation, it occasionally makes arithmetic errors, especially with calculations involving numbers in scientific notation (which it is happy to use as intermediate results if you provide a prompt with a complex, multi-step word problem).
When it can't find the pattern it starts "making things" up, that's where all the "magic" disappears.
It doesn't though. Here's GPT-4 completely failing: https://gcdnb.pbrd.co/images/uxH1EtVhG2rd.png?o=1. It's riddled with errors, every single step.
You are (naively, I would suggest) accepting the LLM's answer for how it 'does' the calculation as what it actually does do. It doesn't do the calculation; it has simply generated a typical response to how people who can do calculations explain how they do calculations.
You have mistaken a ventriloquist's doll's speech for the 'reasoning' of the doll itself. An error that is being repeatedly made all through this thread.
Keep in mind GPT 4 is multimodal and not just matching text.
Sorry for appearing to be completely off-topic, but do you have children? Observing our children as they're growing up, specifically the way they formulate and articulate their questions, has been a bit of a revelation to me in terms of understanding "reasoning".
I have a sister of a similar age to me who doesn't have children. My 7 year-old asked me recently - and this is a direct quote - "what is she for?"
I was pretty gobsmacked by that.
Reasoning? You decide(!)
The robots might know everything, but do they wonder anything?
Machines will have to wonder if they are to improve themselves, because that is literally the drive to collect more data, and you need good data to make good decisions.
There's really no reason to doubt the legitimacy here after everyone shared similar experiences, you just kinda look foolish for suggesting the results are faked at this point.
I once asked my niece, a bit after she started really communicating, if she remembered what it was like to not be able to talk. She thought for a moment and then said, "Before I was squishy so I couldn't talk, but then I got harder so I can talk now." Can't argue with that logic.
Pattern matching? You decide
They may have equivalences, but they're separate forms of mathematics. I'd say the same applies to different algorithms or models of computation, such as neural nets.
I don’t have the language to explain the difference in a manner I find sufficiently precise. I was hoping others might.
Because it's very good at it, sometimes it can fool people into thinking there is more going on than it is.