For example a problem might be solving for the roots of a polynomial, and on each try it would randomly generate a new polynomial with new coefficients.

first problem: find the roots x^2 + 2x + 1

second try: find the roots of 4x^2 - 2x + 4

Or something like that.

Ah, so what you're saying is that they would give up on the problem once they saw the answer, and then move on to a new one, rather than work through the first one until they understood the answer, right?

Sort of. The software would give you several attempts in order to get points on the problem. So they didn't really give up so much as they never had any intent to solve the problem in the first place, so much as see whether there was some kind of obvious relationship between randomly generated coefficients and the answer in order to get points on the question.

Ah, I see, sort of gaming the test platform (or trying to) rather than actually understanding the math. So a case of "you get what you measure" and also an example of what happens when you force kids to learn something they have no interest in, perhaps?