I spent some time in middle management (first as a lead, then as a manager of a small team of devs) and I refuse to do that again. You're 100% right, as you move up, it gets more miserable. As a lead, I was constantly doing adult daycare and counseling for the developers. At the same time, upper management was constantly breathing down my back to improve my teams analytics and performance - regardless of how well the team was doing.

The worst part of the two gigs I had was the director who was in the position above me? Not moving. In both companies. I spent almost six years between two companies trying to find a way out of middle management. I finally realized a lot of companies structure gets flat very quickly because a lot of the upper level executives are not moving, or not getting promoted, so then you have another set of issues you're dealing with.

Its just not worth it unless you can get in and get out of middle management in less than 2 years.