When we look at e.g. higher education, for a place that's particularly guilty of this:
Administrative costs have grown to be approximately equal to instructional costs at universities, from about a quarter of expenditures in 1980.
In essence, our desire to prevent waste and to ensure that money is spent the way we'd like has created an ever-ballooning set of administrative work. It might be better to tolerate a somewhat greater share of instructional and research expenditures being wasted or spent in non-ideal ways.
Any time there's a small bit of waste or money spent in ways we don't approve of, the impulse is to create rules and administrative procedures to prevent it from happening again. But this effectively creates a perpetual overhead for what may have been a one-time, very minor problem.
Most of the parts of the bureaucracy that I think are useless are the parts that are non-negotiable from a legal perspective.