In a mathematical sense, special relativity is not particularly difficult. Undergraduate physics students often learn it by year two as it doesn't require much more than calculus and linear algebra. That's not all to downplay Einstein's genius. If it weren't him, I'm sure others would've gotten there eventually, but he was the first there, and his insights were primarily physical (assuming both the constancy of the speed of light in all reference frames and that the laws of physics are the same in all inertial frames of reference).

General relativity was much more difficult. It took Einstein about a decade to develop it and he had to learn differential geometry in order to do so. This work was undoubtedly more profound and required much more advanced mathematics merged with deep insights such as the equivalence principle [0]. This was what made Einstein so successful: the combination of mathematics with physical insights that no one else had put together at the time.

[0] https://en.wikipedia.org/wiki/Equivalence_principle