Any company of a certain size must convert to a top-down management infrastructure. I've seen what happens in large companies that don't -- you get 15 teams building different versions of the same product, and internal politics decide whose version becomes the 'official' one. But invariably, the internal user community settles on the one that's actually best for their use case, so you end up having to support 3 or 4 versions of every product you create. It's an absolute support and integration nightmare.

You talk about a business like it's supposed to be a democracy. It's not. You need clear direction, and democracy is messy and opaque if your company is too big for you to know the name of everyone who works there. Now the way you describe top down management is a bad implementation of it -- executives need to work to develop their middle managers -- but every company eventually has to convert from a democratic decision making process to a chain of command.

I've been working recently with a number of household-name startups that are having this exact issue. They've hit a plateau in their growth because the culture is dead-set against top-down management, but the company is now too big for people to self-organize. Engineering initiatives that would have taken 2 months in startup mode now take 2 years because they have to go back and rebuild half their systems to make them flexible enough to support the multiple use cases that they now have. Or because they were a startup, they didn't worry about geographic redundancy because that wasn't part of the MVP. So it was put off for later, and now they're a billion dollar company operating out of a single datacenter because their databases and applications were built without multi-datacenter being a requirement. Product can't even reliably tell engineering what features to implement because nobody on the product side has enough authority to shout over the crowd.

So having an infrastructure that cares about feedback is good, but the infrastructure also needs to empower people to do something about said feedback.

I know, I really understand what you're saying. And I am not saying a democracy is a better option. But I just don't think the things you're describing are attributable to top-down management. It certainly has never prevented the problems you've mentioned in any of the places I've ever worked. I agree that clear direction is needed, but your garden-variety middle manager isn't going to provide that. What he will usually provide is a sense that the failures of the organization are the fault of the leaf-nodes.

For every SV-brand startup, there are dozens of small-time consultoware shops serving their local market. They're the kind of places that have been around for 20 years and still only have 100 employees each. They can't ever seem to grow beyond that size because the one-way communication flow from sales to development keeps them in massive technical debt.

Middle managers don't provide direction; they are communication points. As to how effective they can be, it all depends on how the org is structured. If it's a product-centric organization (i.e. P&L rolls up under product and operating costs are charged back to the product org) then it can work pretty well. If it's a functional organization (revenue rolls up through a sales org on a regional basis and is combined with costs at the corporate level) then there's really no amount of management that will make it a good place for developers to work -- under the latter model, finance is running the company and you're a pure cost center. The manager's primary focus will be cost control, not product quality. That's a bad situation to work under.

I agree with your second assessment, but it's more of an issue with the consultoware business model than anything. Most consultoware shops start up thinking they're going to get hockey-stick growth, but it turns out every implementation is super-custom and it ends up as a consulting business. Many companies fail to realize this has happened, and end up operating like a software company when they should be operating like a professional services company with hourly billing. Under a professional services model, if a customer wants a feature, you can quote them a price for developing and implementing it. Tech debt gets dealt with because it becomes a bottleneck for revenue growth when you can't charge clients to add functionality.