I think the idea that brilliant minds have been 'wasted' on prime numbers is nonsense. Don't 'highly intelligent people' have the right to pursue what interests them, and even disregarding that, won't they do their best work on problems that interest them?
Even further, is learning anything that is not practical or useful a 'waste'? Certainly not. Calculus might not be of the utmost importance career-wise for an aspiring musician, but learning it helps us think in new ways.
> The vast majority of "useless" mathematics really do turn out to be useless.
That's fine! So long as we strike gold every once in a while (cryptography, which is pretty essential to the internet functioning as anything more than a bulletin board), math is doing it's job.
That's the worst example you could find, because Einstein didn't develop the mathematics for general relativity. He relied on the math invented in the XIX century for non-Euclidian geometry. If nobody had though about such a "sillY' geometry with "no practical value" it would probably take much longer because the necessary results would be out of the reach for Einstein.
Einstein was also definitely familiar with the work of Helmholtz, who did some fascinating work on non-Euclidean geometry in the context of ophthalmology: Lenses change the amount of curvature we perceive in space (think of fish-eye lenses), and provide a great jumping off point for the notion that the universe might not be as flat as it appears.
The Dover book 'Beyond Geometry' collects a bunch of the major papers in non-Euclidean geometry leading up to relativity, and is a fantastic read.
It's true it probably would have been much harder for him to work out the math on his own. But he and/or others eventually would have done it.
Consider this: a vast majority of mutations are useless, but for this reason, if there were no mutations at all, and mutations somehow willed themselves out of existence, then there would only be primitive lifeforms on earth.
Cryptography is a pretty big deal, though. You can't run a modern economy without it.
The vast majority of "useless" mathematics really do turn out to be useless. In the rare exceptions, there's not much evidence that doing the work beforehand is actually an advantage. E.g. Einstein wasn't aware of most of the work on non-Euclidian geometry before developing relativity IIRC.
Stuff like prime numbers have eaten up millions of brain hours of highly intelligent people. I remember thinking it was weird that so many project Euler problems were about prime numbers. And I looked up what the applications of them were and couldn't find anything significant beyond cryptography.
And they seem to have been chosen for cryptography simply because it was a well studied problem with certain properties. Not because cryptography inherently needs prime numbers and would be impossible without centuries of previous work studying them.