My experience has been that those who are too precious about the only role they've ever had (an "engineer") are only difficult to manage because of their inflexibility and lack of curiosity. It's ironic isn't it?
It's not as absolute as you make it sound. Also because managing a team shouldn't require you to make very technical decisions. You should obviously know what you are managing but if there is a decision to be made, the people who spent hours on that subject will be your best experts and their guidance is essential.
In soccer coaches usually handle the strategic side of things and they have dedicated coaches who handle more player-related things. I think this is how a good manager in a normal company should work too. Make sure the team works as a team and take strategic decisions but not handle the nitty-gritty technical stuff.
a) are very bad at knowing how to measure performance
b) mostly only measure performance as an individual contributor, and not as a manager or leader. (Which is to say: of course managers and leaders have their performance assessed, but it is often via the same tools/questions used for individual contributors, rather than specifically measuring skills management or leadership domains)
Then you overlay this with human factors such as nepotism, conflict aversion, politics, friendships… and of course we screw it all up overall.
—-
So, to your point, absolutely we’re bad at selecting people who have the aptitude to be a good manager or leader for such roles, and terrible at weeding out people who aren’t suited.
I would encourage you to try to setup a place where no hierarchy exists and see how it goes. In my experience people have wildly different opinions on multiple (even minor) things and without someone to "take a decision", things end up just going in circles.
Of course, if the person taking the decisions repeatedly takes decisions that have a negative longer term impact (much easier to asses on longer term than at the moment you have to take the decision), something should change, but that can include contributors looking for other companies ...
This is a point I came to make. Different jobs also different skillsets. A good solution taken on by some industries is decoupling promotion and salary rises by having separate engineering and management streams where both have similar opportunities for career progression but you can stick to the actually job that you yourself are best suited for. People who have the people/social/political/organization/high-level-view skills and like exercising them can go into management, and those with focus/deep technical expertise/creative problem solving skills, can stay in engineering. People do usually have bits of both of these, but you know if you are stronger in one or the other.
Senior engineers and above, they need to know all the engineering skills, but they are most of their time not being engineers, they're being a people person. They're talking to others, they're explaining to other developers what's going on, how to do their job. There is somewhat difference as the manager is thinking only about the organization structure and how good people are, whereas the senior engineer is thinking about how to use people to make the engineering parts better.
Both is fine, these are just different jobs. However, the problem is that the manager is 'above' the engineer in the generally accepted hierarchy. Consequently, engineers are pressured to become managers for the development. As long as this hierarchy exists, the problem will exist.