In the 20th century it became trendy to (a) try to base everything on set theory, (b) write mathematics in a very dry and formal style, taking inspiration from the Bourbaki project.
My impression is that undergraduates are taught in the language of sets partly because it is serviceable for describing the subjects they are trying to learn (esp. analysis) but also more importantly because mid-20th century mathematicians who set up the curriculum we are still using wanted to thoroughly teach (indoctrinate?) undergraduates the trendy style of the time.
Personally I think it has been a mixed bag; the style is a big turn-off to many people, and ends up chasing people out of mathematics who could otherwise make valuable contributions. The people who remain seem to mostly like it okay though. YMMV.
First, why the scare quotes around "rigorous"? Set theory literally did make the foundations of analysis more rigorous.
Second of all, we don't use set theory because it's "trendy" or because we want to "indoctrinate" undergraduates, we use it because it's the best known foundation for mathematics. If there was something less complicated or annoying that worked, we'd use that instead. But as far as we know, there isn't, and there are good reasons to believe we won't find something better.
But I've come to understand that I was not realizing that I didn't like mathematics or thought it was too hard, but instead that if I made that my life's work, I'd be working with many more people like that, rather than the people I wanted to - bright and creative, yet kind and humble.
I don't think set theory or any other set of tools is the problem behind potential academics getting turned off. It's the people, and the culture of glorified monkhood.
It is possible to use categories as basic building blocks instead if sets but, in my anecdotal experience, this us not what the majority of graduate programs in Mathematics do.
It will be interesting to see whether this will change in the next 20 yea.
This issue has been litigated extensively, and in my view successfully, by Harvey Friedman on the Foundations of Mathematics mailing list, if you want to check its archives.
The vast majority of mathematicians go their entire lives without using the word "category" in a paper. What are they missing out on?
One amusing problem is discussed here: https://mathoverflow.net/questions/289711/defining-sun-in-ho....
But suppose HoTT were equally good. What compelling reason is there for a working mathematician to learn it? We already know about set theory, and it meets all of our needs.
I also think it's important to point out for anyone reading that one certainly doesn't need category theory to do proof verification. Some other options, and some gripes with the Coq community along the lines of the previous paragraph, are listed here: https://xenaproject.wordpress.com/2020/02/09/where-is-the-fa....
Also, if your foundational theory doesn't allow me to (easily) define a fundamental geometric object like SU(n), then it's just a non-starter. Again, I have not been able to find a reason why we ought to endure such pain to do simple things when we already have a perfectly good foundational theory. Re: the HoTT book, it is precisely because I've looked at the book that I have these questions.
Most undergraduates learn mathematics in the language of sets. If they're doing algebra, then the underlying structures of their groups, rings, etc. are always sets. Same for topology and logic (e.g. classical model theory). An extreme example of this set-theoretic thinking is defining the real numbers in terms of Dedekind cuts.
Category theory, very slowly, changes this perspective. If you study this for years, you will come to realise there is more to mathematics than sets. And your way of thinking will shift. You will learn to ask different questions. For example, instead of defining things in terms of their underlying set (often breaking symmetries), you will ask 'does it have a universal property?' and so on. Note, there are still tons of reasons to use sets / ZFC, even when working with categories mostly. But I do not want to get into that and it's irrelevant for the point I'm trying to make.
Final note: the 'language of sets', i.e. the language of undergraduate mathematics, is very different from set theory. 'Set theory' is like the language talking about itself. The same goes for pure category theory.