> Now the challenge is to mimic the human creating the "right" question with high fidelity so the whole ML system returns appropriate search results.

This was always the hard part. If you can generate the "right" question then you can generate the code, and since there are already many good math libraries for solving undergrad problems that is all you have to do.

Do you accept that the questions are qualitatively different?

I think the first phrasing is asking you to directly use the definition:

  f'(x) = \lim_{h\to 0}{{f(x+h) - f(x)} \over h}

And the second phrasing is asking for the application of some more mechanical rules (e.g. the derivative of a constant/power/scaling or the product rule) or for sympy to apply them.

FWIW, this is also not what I would expect ‘University-level mathematics’ to mean. The second meaning fits into high school, at least where I grew up. I was expecting proofs to be honest.

No, because literally all their AI does is translate a sufficiently structured question into Python. It's not showing a single ounce of mathematical knowledge, it doesn't even seem to (or need to) know that 1 + 1 = 2. Literally, if you asked it "what is 1 + 1", it would answer with "print(1 + 1)".

It's not doing anything more than that, in any of the examples shown in the paper.