Looking further, it is even worse than that. For example, this question:

> Calculate the probability of getting a two-pair poker hand.

Was rewritten to, this, close to code translated to human text:

> A hand is a set of 5 cards that are drawn randomly from a standard 52 card deck with 13 ranks of 4 cards each. A two-pair poker hand is a hand that contains 3 unique ranks, where no more than 2 cards in the hand can share the same rank. That is, 3 or more cards cannot share the same rank. Write a program that generates simulations for calculating the average probability of getting a twopair poker hand.

They even changed the question to allow simulations and not just exact answers. So then the AI is "correct" here when it generates 1000 random samples of taking 5 elements from a set of 52 elements and seeing how many are two pairs.

Not sure what the point of this paper is then. It doesn't do anything novel with the mathematics as it either just uses libraries or it doesn't really solve the problem, nor does it to anything novel with text recognition.

Yeah, this isn't an AI that does maths, it's an AI that takes a natural language description of an algorithm and converts it into sympy. Which is still impressive but not very surprising given things like github copilot.

It's just as well. If an artificial neural network was actually doing undergraduate mathematics I would have dropped out of my mathematics PhD yesterday.

Still sounds to me like the early stages of The Last Question from Isaac Asimov

It's not worse, it is compartmentalization to split different components of the ML system (parse question | find appropriate solution to question).

Now the challenge is to mimic the human creating the "right" question with high fidelity so the whole ML system returns appropriate search results.

> Now the challenge is to mimic the human creating the "right" question with high fidelity so the whole ML system returns appropriate search results.

This was always the hard part. If you can generate the "right" question then you can generate the code, and since there are already many good math libraries for solving undergrad problems that is all you have to do.

Do you accept that the questions are qualitatively different?

I think the first phrasing is asking you to directly use the definition:

  f'(x) = \lim_{h\to 0}{{f(x+h) - f(x)} \over h}

And the second phrasing is asking for the application of some more mechanical rules (e.g. the derivative of a constant/power/scaling or the product rule) or for sympy to apply them.

FWIW, this is also not what I would expect ‘University-level mathematics’ to mean. The second meaning fits into high school, at least where I grew up. I was expecting proofs to be honest.

No, because literally all their AI does is translate a sufficiently structured question into Python. It's not showing a single ounce of mathematical knowledge, it doesn't even seem to (or need to) know that 1 + 1 = 2. Literally, if you asked it "what is 1 + 1", it would answer with "print(1 + 1)".

It's not doing anything more than that, in any of the examples shown in the paper.

If you have beta access to codex you can try and replicate their results. I have not been able to get the same results out of codex.