I think mathematics is in a different class than physics or other scientific endeavors because there is meaningful mathematics going back to ancient Greece whereas I am not aware of any physics that was known thousands of years ago that is still relevant.

So does it mean we should curb some superhuman thinking and let them explain better to other human being? I'd think so. It is all about balance or trade-off, but at some point, a field would become too esoteric for a society and it might need to slow down the short-term progress to ensure consistent long-term progress.

It seems the math field as a whole (or more specifically the number theorists?) needs more of good communicators who can link the cutting edge and more general public. The math academia could give incentives to such activities in addition to pushing the boundaries.

The problem is that things genuinely are hard, and there's only so much compression you can do before you're reduced to meaningless handwaving. Textbooks are already compressed by a factor of 100 relative to the original research literature, sometimes even oversimplified. Demanding something a factor of 100 better than a textbook is going to get you something like one of those useless "learn C++ in 24 hours" paperbacks.

Papers are not rewarded for being easy to understand. People/organizations get rewarded for publishing "complicate" papers. That misalignment is the issue.

For state-of-the-art research in obscure specialisms, of course you're right.

However, numerous papers and books cover more mundane subjects and there is no good reason they could not be readily accessible to anyone with an undergraduate-level background or at least a masters delving a bit deeper into the field of interest. Often the problem isn't dumbing down the material, it's simply poor communication skills.

Put another way, while our cumulative understanding of mathematics now covers a very broad range of areas, that doesn't necessarily mean the depth to reach a good understanding of any particular specialism has increased at the same rate. On the contrary, given that even the most specialised of theories must be something that an individual can come to understand and build upon within a single career, there is an inherent limit to how deep our understanding of the subject can go. That limit fundamentally depends on how efficiently we can build the layers of more elementary understanding on which the start of the art must rest.