But still, it does not really matter. A LLM, without harness or tool, cannot do math: it returns answers that are "reasonnable guesses based on historical data" without knowing which ones are non-sense or not, and it needs tools and harness that "do the math" to rejected the non-sense.
> You need to understand the problem to know where to search for the solution.
"What you mean by understanding"
you: "well, that thing that LLM does"
"what's the thing"
You: "Understanding!"
"And what is understanding?"
You: "It's the thing that AI do! I just said that!"
But, no, the "logic", the "structure", of math, is not inside the LLM. The LLM regularly returns outputs that are very good, because turns out that some relationships that the LLM extracted from historical data was also the relationships that happen to work here. And sometimes, the LLM returns outputs that are totally crazy, incompatible with "understanding the problem". Because the LLM is just doing "educated brute force" and applied a set of relationships that someone who is familiar with the problem would immediately consider inadequate, but the LLM still does it because it does not understand the problem, just tries to guess the direction of the answer based on historical direction of answers, without any idea if these directions are non-sense or not.
You're trying to argue that a Michelin Chef doesn't understand cooking because she only uses techniques and ingredients that others have used before - even though the dishes are totally new and other chefs are studying her techniques.
It is still possible that I'm incorrect, of course. But my understanding of how it works seems to fit very well the data.
"generalization of distilled knowledge" is a naive way out. People who have experience with ML have seen how difficult it is for the system to not get stuck in a local minimum or not latch on to a spurious pattern. Just for arithmetic, providing tons and tons of 6 digits multiplication tables, it is still extremely difficult for a ML model to extract the knowledge, there are a huge amount of fake patterns that leads it to incorrect heuristics. For example, the values of the first digit of the two numbers are strongly correlated to the value of the first digit of the answer, so the training will push into the direction for the model to learn to answer the computation based on this kind of heuristic. The "distilled knowledge" that the LLM will extract is not basic mathematical rules, it is "74% of the time, it's 4 and 26% of the time, it's 6", and it will fall into this local minimum and it will be difficult for it to get out of it.
But as I've said, I'm open to the possibility I'm wrong. If it's the case, maybe you can convince me otherwise. But it would help to bring concrete arguments more than "you sound like you don't have experience on this subject".
> You're trying to argue that a Michelin Chef doesn't understand cooking because she only uses techniques and ingredients that others have used before
But I don't say that LLM cannot do math because they use techniques and ingredients that others have used before. I'm saying that LLM cannot do math because the way they work. Fundamentally. The LLM extracts pattern by minimizing the loss function. There are hundreds of heuristic that will lead to a small loss function while still returning bad answers. Some patterns are "the good ones", but hundreds of them are misleading ones. And they can even get stuck in the wrong valley, with the way they learn preventing them to climb the ridge to join the "correct" valley. The fact that LLM that were very convincing to talk like they understand the human language and yet had a 2% success rate with basic arithmetic shows that LLMs often fail to extract the correct patterns. These LLMs did train on tons of numbers. To the point that they were able to do very convincing things. And yet, they did not discovered the "distilled knowledge" that were the arithmetic rules.
Perhaps you picked up the belief that LLMs are just brute forcing these problems, but they aren't. They are creating hypotheses and checking them, which is the same thing humans do.
The solution spaces for problems is so massive that brute forcing it is beyond what even "every atom in the universe is a super computer running for 13B years could crack"
You need to understand the problem to know where to search for the solution.