As a rough generality, being good at something, like proper good at it, is incompatible with the basic forces of how modern organisations work. By this I mean the systems of reward, power, decision-making.

That doesn’t mean all organisations work this way (mythically, startups of old didn’t), nor that some leaders can’t manage this tension, but tension it is which means while these leaders will often achieve “better” outcomes (that is, “more good” outcomes with relevance to the area of expertise or craft they’re good at, often but not always with more empathy and better working conditions for those delivering), they’ll also encounter more barriers to progressing to and maintaining positions of power, face greater scrutiny, probably be confused why the organisation around them isn’t understanding or valuing their capability (and spend a lot of time translating to frames they do understand and value), and ultimately greater burnout.

I think the challenge is largely a matter of scale. Once you have thousands or tens of thousands of people, communication becomes by far the dominant factor in outcomes. You can set up different structures and cultural norms to try to nudge things in the right direction, but there is no org structure that can solve this because local details always matter. The trick is understanding which details matter and why.

Ultimately I think this depends on management competence and judgement. I also think IC leadership is a critical counterweight to the distortions of empire building that incentivize creating messes. And finally as a competent individual you need the maturity to recognize the difference between unavoidable but tolerable organizational dysfunction, and broken leadership where its impossible to do good work, and hence time to cut bait and leave.

Mediocre developers are the first to hop over to the management track or non-coding sanctuaries like system architecture. So naturally they are over represented in tech leadership.