human wrong => no problem, mistakes happen
Practically for outcomes it doesn’t matter if that learning is happening in the LLM or on a layer above it.
But 17, 37, 47, 67 and 97 are primes, so he probably applied a pattern subconsciously because you would never say 27, a drill cube, nor 77, which is divisible by 11 on sight.
I've taught mathematics at assorted levels, I think I can tell the reasoning behind quite a few human mistakes there. I analyse them, they're not usually just whatever, there's a wrong chain of thought behind the interesting ones. That's how I design exercises, tests and generally teach stuff to humans.
So I've been doing this mostly within Skype and the thing doesn't err because it's reasoning wrongly, it errs because it just comes up with stuff, there's nobody home. And that would be my professional assessment of it as a student. And yes, quite often it gives you good answers for the initial base line questions, but if you refine the problem to probe its level of "understanding" it's not difficult to derail it, it elaborates on its previous procedures while it's being steered towards a wall of nonsense.
https://en.wikipedia.org/wiki/Braitenberg_vehicle
Samuel Johnson: “Depend upon it, sir, when a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully.”
Also mathematics is not the right field to compare this to because there are rules. In languages and object recognition/synthesis it is all subjective. Understanding here means understanding context and human-subjective interpretation.
When I say "that sunset is beautiful" you understand what I mean, ML models simply approximate based on what they see other humans do or say. I am not calling the sunset beautiful because other people are, it is my own subjective interpretation.
Alexander Grothendieck, one of the greatest mathematicians of the 20th century, answered "57" when asked in a public lecture to provide an example of a prime number. 57 = 3 * 19 is not a prime number.
According to your argument, this would imply Grothendieck did not understand what a prime number is. Which is laughable.