We've all what a terrible job LLMs do in some problems. But they are doing well at burning down Erdos's list of problems. Perhaps mathematicians aren't a good measure of "truly smart".

If you're seriously taking that angle on comparison .. how well are they doing on generating interesting and challenging Erdős like problems?

Oh, not that seriously, but it makes me wonder!