Obviously there's other factors (effort, as you point out).
Though I think it's partly an artifact of the days when an undergrad degree would bring students to the leading edge of research, so there was simply no-one else qualified. There would have been a lot more synergy when researchers were explaining cutting edge ideas to students (it still happens a bit, but it's the exception rather than the rule).
Yes, that was my suspicion too -- historically, students were only going to those lectures where they were actually interested and which they were much closer to the level of the lecturer. In that case, the bottleneck isn't "ability to be a good explainer to novices" but "ability to answer arbitrary questions".
IOW, it looked more like grad school today, where the good researchers naturally are a better fit for teaching, and don't mind it as much.
They aren't. To an ever increasing extent, undergraduate teaching is done by adjunct staff, who are not involved in research, and who work on a temporary basis.
Professors with good research funding can often buy themselves out of all or part of their teaching load. Many choose to teach only the upper level or graduate courses, where immersion in research is in fact a useful qualification for teaching.
Disclosure: I was an adjunct for one semester, many years ago.
Meanwhile, tenure-track/tenured professors tend to focus primarily on teaching upper-level classes, which suits them infinitely better. The students they get are much more knowledgeable, which frees profs from worrying too much about background knowledge; by sheer virtue of selection bias, students that aren't interested have been filtered out; and the material is much closer to the work the profs themselves actually do.
[1] https://www.theatlantic.com/business/archive/2013/04/the-eve...
For lower level subjects (i.e. most undergraduate courses) this is much less an issue, but the above still explains why we need researchers to also teach. Issues occur when you let a brilliant but eccentric mathematician teach calculus 101. His research is in no way relevant to the course, and he might not be able to explain things on that base a level.
Sadly, sufficient mastery of the subject often comes at the cost of sufficient mastery of teaching, which becomes especially important for such difficult subjects. If only some of that money spent on more staff was spent on assistant teachers whose main skill was teaching and coaching other teachers, and providing paid hours for both of them to fix the curriculum together...
Anyway, I fully agree with your second paragraph. Another issue is that of researchers not really seeming to care, having an attitude of "you shouldn't be in University unless you really want to learn this, and I'm not going to try to motivate you or help you with that." There's some truth to that, but it also sometimes feels like it's used as a way to mask their own teaching incompetence.
I was asked to be an assistant teacher after graduating from my masters, and I completely freaked out about the responsibility of making sure I wouldn't fuck up my student's education (I cared more than most of my students did). I almost declined the job offer! I was basically told "yeah we know, that's part of why we want you."
Anyway, I spent a lot of time on-line trying to find advice on the dos and dont's of education. I can say that the best two lecture were Eric Mazur's Confessions of a Converted Lecturer[0] and John Corrigan's Are We Listening To Our Children?[1]. I think every teacher should watch these at least once.
I'm mostly thinking about the highest level math courses though.
There's a meme in research that teaching helps you do better at research. The story sounds plausible: to do a good job at teaching, you break the subject down into its component parts and tease out the essence, then package that up to deliver it. Perhaps through this process of repeatedly breaking your subject down to teach it, this process will eventually lead you to think about something differently, and then maybe you get an idea for your research. I've talked to people that say they have had this happen to them.
Partly because the idea that teaching is hard is heretical, though, professors really de-prioritize teaching when they are mentoring new professors. So not much in the way of classroom skill gets passed down from generation to generation, and it's up to the individual professor to re-discover how to teach when they start their career. Outcomes vary. Additionally, because of the idea that teaching is not hard, new professors (and professor trainees) are discouraged from spending too much time on teaching, so in my experience, when it's done well, it's done either by someone that takes more time than they ought (potentially at the expense of their career) or someone that is just so great at everything they do, that it works out okay.
This means there are just not that many people floating around who know how to teach and are amped up to do it. You see this when you try to staff "lecturer" positions at US universities, where that term means "someone who teaches but does no research." Those positions might even come with their own analogue to the tenure track! And you can interview dozens of people that can't really lecture, or teach.
If you go to grad school, you get indoctrinated very quickly to the idea that teaching is easy, you should do it with your eyes closed, but you should put in the minimal amount of work required before doing something that really matters, like publishing papers. I think it's hard to find people that can teach because those people wash out of grad school very early when they find out that their attempts at teaching well will not be rewarded.
This is not true if you look at the tiny liberal arts colleges, though. They actually care about teaching. However, if you go to grad school, you go to an R1 school (because tiny liberal arts colleges can't afford grad students) and R1 professors think that becoming a liberal arts college professor is a shade of failure, so they will strongly discourage you from going down that path.