By far the most dissapointing thing was that the client often just wanted to go with Google's recomendations or they had even worse ideas. Even when we really tried to look out for the best interests of the client it was like fighting an uphill battle both ways.
Another problem is that anything that isn't direct response is hard to measure so it's tough to make a clear and compelling case. Anybody can chalk up any hare brained idea to brand value and it is tough to push back on that when the client doesn't really care.
However it's worth pointing out that Google as an org have absolutely hated ad agencies for many years, in fact being nice to the agencies was a strategy used by FB when I was there.
I'm not disputing your point about optimisations, but the notion that Google like ad agencies is just ludicrous.
The agency staff can then claim the campaigns were optimized based on Google's suggestions, which is the perfect smokescreen.
Most big brands that outsource media buying don't have SMEs to audit the media buying they've outsourced, so the result is Google makes a ton of money via their preferred middleman, advertising agencies.