It is still possible that I'm incorrect, of course. But my understanding of how it works seems to fit very well the data.
"generalization of distilled knowledge" is a naive way out. People who have experience with ML have seen how difficult it is for the system to not get stuck in a local minimum or not latch on to a spurious pattern. Just for arithmetic, providing tons and tons of 6 digits multiplication tables, it is still extremely difficult for a ML model to extract the knowledge, there are a huge amount of fake patterns that leads it to incorrect heuristics. For example, the values of the first digit of the two numbers are strongly correlated to the value of the first digit of the answer, so the training will push into the direction for the model to learn to answer the computation based on this kind of heuristic. The "distilled knowledge" that the LLM will extract is not basic mathematical rules, it is "74% of the time, it's 4 and 26% of the time, it's 6", and it will fall into this local minimum and it will be difficult for it to get out of it.
But as I've said, I'm open to the possibility I'm wrong. If it's the case, maybe you can convince me otherwise. But it would help to bring concrete arguments more than "you sound like you don't have experience on this subject".
> You're trying to argue that a Michelin Chef doesn't understand cooking because she only uses techniques and ingredients that others have used before
But I don't say that LLM cannot do math because they use techniques and ingredients that others have used before. I'm saying that LLM cannot do math because the way they work. Fundamentally. The LLM extracts pattern by minimizing the loss function. There are hundreds of heuristic that will lead to a small loss function while still returning bad answers. Some patterns are "the good ones", but hundreds of them are misleading ones. And they can even get stuck in the wrong valley, with the way they learn preventing them to climb the ridge to join the "correct" valley. The fact that LLM that were very convincing to talk like they understand the human language and yet had a 2% success rate with basic arithmetic shows that LLMs often fail to extract the correct patterns. These LLMs did train on tons of numbers. To the point that they were able to do very convincing things. And yet, they did not discovered the "distilled knowledge" that were the arithmetic rules.
You're trying to argue that a Michelin Chef doesn't understand cooking because she only uses techniques and ingredients that others have used before - even though the dishes are totally new and other chefs are studying her techniques.