Are you sure you know what math is, or are you thinking about arithmetic?

Tao and others don't sit around working out what 12,483,324,281/188,553.37463286*(243+2-983) is all day. There is a reason most people don't consider mathematics education to really start until you stop needing to bring a calculator to class.

You seems to have scanned the message and notice one word ("basic math calculator") and stopped there.

Just next to this word, there is "Lean verification", which is not about arithmetic but about the logic.

But the problem is not even there. LLM are not able to do math not because they fail to do 12,483,324,281/188,553.37463286 * (243+2-983), but they are not able to do math because they don't _understand_ 12,483,324,281/188,553.37463286 * (243+2-983).

I cannot do 12,483,324,281/188,553.37463286 * (243+2-983) myself just like that, yet I can do advanced math. Because I understand this expression. I know what it means. When the LLM was not able to produce the correct number, it was not because it knew what this expression meant and was "distracted" and forgot to report a 1 when doing the mental calculation. It was because the LLM was seeing the expression and was trying to guess the answer based on the answers of other texts that had similar expressions. It did not _understand_ the expression itself: it had no idea what it meant, it was just guessing the answer by copying the output it has seen in the wild.

And this is why it was a good element to demonstrate that LLM cannot do math: if they don't understand the expression 12,483,324,281/188,553.37463286 * (243+2-983), then they don't understand the expression f(t)=\sum_{k=-\infty}^{\infty} c_k e^{j k\omega_0 t}, \qquad \omega_0=\frac{2\pi}{T}.

And that is why the AI framework to do proper math and help mathematicians needs harness with tools like Lean. The LLM part does not understand what the expression means, it needs a tool to discard, internally, all the LLM outputs that are just non-sense.

"The AI can't understand because AI doesn't understand anything"

So what is understanding then, so we can check this hypothesis?

"It's the thing that AI can't do!"

What's the thing?

"Understanding!"

And what is understanding?

"It's the thing that AI can't do! I just said that!"

FYI, this isn't a serious argument without a clear testable definition of "understand" or "know".

Oh dear, not this again.

First, the claim has nothing to do with "understanding".

The claim is "LLM cannot do math". LLM. Not LLM + tools and harness. Not LLM but we get plenty of answers, run a program to remove the non-sense one and pretend the LLM got only one answer.

And this is true: the LLM part cannot answer these math questions. It answers with a bunch of answers and has no idea which one is correct and which one is non-sense. It needs a specialised harness that will remove the non-sense ones.

But secondly, this incapacity to grasp something very simple is really annoying.

This is really not complicated. If you tell someone "what is 2+3" and that this person is going in the next room, write on their pocket calculator "2+3=1", click "ok", and the calculator says "Error", then "2+3=2", click "ok", and the calculator says "Error", then "2+3=3", click "ok", and the calculator says "Error", then "2+3=4", click "ok", and the calculator says "Error", then "2+3=5", click "ok", and the calculator says "Correct", and come back and say "2+3 is 5", it is clearly not the same as if you ask someone who understand math.

Call it "understanding math", or "able to do math", or "has learnt arithmetic", ... It does not matter. The reality is that this person, ON THEIR OWN, is unable to do math. They are not the one who do the math, the calculator who tells them if it works or not is the one doing the math.

We've had the ability to brute force solution spaces for decades, without AI.

Perhaps you picked up the belief that LLMs are just brute forcing these problems, but they aren't. They are creating hypotheses and checking them, which is the same thing humans do.

The solution spaces for problems is so massive that brute forcing it is beyond what even "every atom in the universe is a super computer running for 13B years could crack"

You need to understand the problem to know where to search for the solution.

They are not "simply brute forcing". They are constructing hypotheses based on their training on other examples. So, it's not a simple brute force, it's an "educated by historical data brute force".

But still, it does not really matter. A LLM, without harness or tool, cannot do math: it returns answers that are "reasonnable guesses based on historical data" without knowing which ones are non-sense or not, and it needs tools and harness that "do the math" to rejected the non-sense.

> You need to understand the problem to know where to search for the solution.

"What you mean by understanding"

you: "well, that thing that LLM does"

"what's the thing"

You: "Understanding!"

"And what is understanding?"

You: "It's the thing that AI do! I just said that!"

But, no, the "logic", the "structure", of math, is not inside the LLM. The LLM regularly returns outputs that are very good, because turns out that some relationships that the LLM extracted from historical data was also the relationships that happen to work here. And sometimes, the LLM returns outputs that are totally crazy, incompatible with "understanding the problem". Because the LLM is just doing "educated brute force" and applied a set of relationships that someone who is familiar with the problem would immediately consider inadequate, but the LLM still does it because it does not understand the problem, just tries to guess the direction of the answer based on historical direction of answers, without any idea if these directions are non-sense or not.

I'm sorry but you are horribly confused about how these models work. This reads like your understanding comes from social media comments (stochastic parrot stitching together snippets from an archive) rather than what is actually happening (generalization of distilled knowledge).

You're trying to argue that a Michelin Chef doesn't understand cooking because she only uses techniques and ingredients that others have used before - even though the dishes are totally new and other chefs are studying her techniques.

Well, I am a scientist with a PhD, I work with ML professionally since "before it was cool", and I have built my own toy LLM.

It is still possible that I'm incorrect, of course. But my understanding of how it works seems to fit very well the data.

"generalization of distilled knowledge" is a naive way out. People who have experience with ML have seen how difficult it is for the system to not get stuck in a local minimum or not latch on to a spurious pattern. Just for arithmetic, providing tons and tons of 6 digits multiplication tables, it is still extremely difficult for a ML model to extract the knowledge, there are a huge amount of fake patterns that leads it to incorrect heuristics. For example, the values of the first digit of the two numbers are strongly correlated to the value of the first digit of the answer, so the training will push into the direction for the model to learn to answer the computation based on this kind of heuristic. The "distilled knowledge" that the LLM will extract is not basic mathematical rules, it is "74% of the time, it's 4 and 26% of the time, it's 6", and it will fall into this local minimum and it will be difficult for it to get out of it.

But as I've said, I'm open to the possibility I'm wrong. If it's the case, maybe you can convince me otherwise. But it would help to bring concrete arguments more than "you sound like you don't have experience on this subject".

> You're trying to argue that a Michelin Chef doesn't understand cooking because she only uses techniques and ingredients that others have used before

But I don't say that LLM cannot do math because they use techniques and ingredients that others have used before. I'm saying that LLM cannot do math because the way they work. Fundamentally. The LLM extracts pattern by minimizing the loss function. There are hundreds of heuristic that will lead to a small loss function while still returning bad answers. Some patterns are "the good ones", but hundreds of them are misleading ones. And they can even get stuck in the wrong valley, with the way they learn preventing them to climb the ridge to join the "correct" valley. The fact that LLM that were very convincing to talk like they understand the human language and yet had a 2% success rate with basic arithmetic shows that LLMs often fail to extract the correct patterns. These LLMs did train on tons of numbers. To the point that they were able to do very convincing things. And yet, they did not discovered the "distilled knowledge" that were the arithmetic rules.