Indeed, but let me ask you this: what real responsibility do all those layers of middle management in a typical big software company actually have?
Detailed planning of individual features is junior management stuff. The only estimates that are really worth a damn come from the junior managers, based on the technical data provided by their teams.
Strategic product and project management -- what products and major functionality are going to be built and when they will ship -- is senior management stuff. The technical decisions that really matter are taken by senior managers or executives. Are we going to drop major projects W and X but include Y and Z, given that this means we can ship a new version of this product in Q3? Shall we authorise a 20% increase in payroll budget for this product's team, given that this would allow us to include project Y by that deadline as well?
Obviously in a large organisation with many products, each of which is itself large, there is scope for having layers of management in between so everyone is working on a sensible scale. But you can scale out a pretty long way with just a single layer containing just a few extra managers, and I'm pretty sure that the 2/3/4/5/6 layers you often find in a lot of big software companies are mostly dead weight.
Firmly agree and I'll go a step further. I think the traditional org structure and traditional titles aren't well suited for emerging technologies.
Fact is, whoever is held responsible for succeeding/failing is going to get paid more. Being held responsible sucks and is hard.
I think a reasonable alternative would be smaller, largely decentralized teams that don't have traditional management, instead having two people they interface with who take over the traditional management roles - a logistics/supplies guy who checks what each team needs and makes sure they're equipped, and a planning/coordinating guy that makes sure the technical team has something sort of resembling a plan that fits in with the organization's grand strategy.
Team size would be 3 to 10 people or so, with one logistics/supplies guy and one planner/coordinating guy checking in with 5-10 teams at regular times and being on call during certain times if needed.
Something like that. As long as you've got a manager and he's held accountable for results, he'll have higher social status, be more neurotic, and expect to be paid more. I think moving to a supplies/logistics guy (similar to a servant manager type) and a planning/coordinating guy (strategist/diplomat/mediator) you could build an organization where engineers had more respect and authority and seniority, but things would be kept in check. There'd still be some friction between your planning/coordinate guy and technical people, which is natural, but ideally less so and there'd be more of a push for good engineers to lead a new team working on a new project than to become middle managers/planning guys. Largely different skillsets, engineering and planning/coordinating. Google Wave is a pretty good example of why you need a planning/coordinating guy and not just a pure engineering project, but engineers should have the most authority, social status, and prestige in a technology organization.