You are correct in that most companies are not equipped to staff issues like this. Most places just accept their bills as a cost of doing business, not something that can be optimized.
In reality, you want both. A good systems person can save you a ton of money.
Seriously, some employers I’ve know seem to take exactly that approach. Just fire all the ops guys, and tell the devs that they’re now going to be doing DevOps.
The issue is that once you hire a dozen engineers to do this (say for 5M a year in total), and they do it for a year, they save mid 8 figures (keep in mind this was the largest service, so the savings across other services will be smaller).
Then can they keep saving mid 8 figures every year?
I'll paraphrase something I previously wrote privately, but imagine you have some team that's able to save 10% of your fleetwide resources this year. They densify and optimize and improve defaults. So you can now grow your services by 10% without any additional cost increase, and you do! The next year, they've already saved the easiest 10%. Can they save 10% again? Can they keep it up every year? How long until they're saving 3% a year, or 1% a year? And that's if you keep the team the same size, where its clearly loosing money! If you could afford a dozen people to save 10%, you can only really afford 1-2 to save 1%, but then you're likely to get an even smaller return.
Unless you expect to be able to maintain the same relative value of optimizations every year, 3 or 5 years out, its not worth it to hire an FTE to work on them.
I should note that I've experienced this myself: I was working in an area where resource optimization could lead to "significant" savings (not 8 figures, but 6 or maybe 7). My first 6 months working in this area, I found all sorts of low hanging fruit and saved a fair amount. The second six months, 5-10x less ROI. I gave up even trying in the third six moths, if I come across a thing, I'll fix it, but its no longer worthwhile to look.
If you're looking across the business as a whole, it seems likely that there is a lot of this kind of work lying around because there is not much incentive for people to tackle it as described in this comment: https://news.ycombinator.com/item?id=29691847.
The issue is still the same: what do you do in the second year (or the third), after you've fixed all the low hanging fruit. If it takes a constant amount of time to investigate a particular service, it's not even worth examining the long tail, because the investigation is costlier than the savings. So once you've saved the high 8 figures by fixing the low hanging fruit in the biggest 20 services or whatever, what next?
The broader point is that while it is often very worthwhile to have individual employees work on optimizations, it is much less often worthwhile to task teams entirely with optimizations, especially in the way you describe. Having some group in charge of generalizable fleetwide optimizations is possibly useful. Doing some kind of resource quota thing where you get fewer resources than you're forecast to need (to force you to do some kind of local optimization) may make sense, but having a strike team whose job it is to tune JVM parameters isn't useful. A central team writing a document on best practices, setting better defaults, or using/building some kind of grid-search optimizer are all probably better investments.
Good large engineering organizations have these teams, and understand their value. For smaller companies, it's not as clear if you can reasonably fund a team for that kind of work.
Then you destaff the team, collect your tens of millions of dollars, and be happy about a successfully finished project? The problem with going too far down this line of reasoning is that you leave even low-hanging fruit laying around forever.
Like I've said twice now, you can solve this problem without stupid business practices, and the you're arguing with a straw man. If you have to say "going too far down this line", you're no longer responding to the argument presented, but a bad faith misrepresentation. That's inappropriate on HN.
In my team, the key is for the state of the project/product we manage, cost optimization is likely one of the lowest ROI activities we can spend too much time on. That doesn't mean we don't tackle some clear low hanging fruit when we see it, or use low hanging fruit as training opportunities to onboard new team members, but that we need to be conscious on where we make investments, and for the stage we're at, the more important investment is into areas that make our product more appealing to more customers.
I think it's easy to say someone, like an intern could pay for themselves with savings. But this to me overlooks that someone has to manage that intern, get them into change management, review the work, investigate mistakes or interruptions from the changes, etc. And then they're still the lowest earning employee, since most of us aren't hired to pay for ourselves, but actually to turn a profit for the company.
So while I'm not sure I agree with the message "whereas increases in income are larger and compound better.", I certainly understand and have pushed a similar message, that we be conscious on where we're spending our time, and that we're selecting the highest impact activities based on the resources and team we have. Sometimes that may be fixing high wastage, but very frequently that will be investing into the product. And I think for the stage of the product we manage, that is the best choice for us.
Result is that maximising income is generally better than reducing cost. Of course, as with all generalisation, there are situations where this approach doesn’t hold true. But as a high level first order strategy, it’s a good one to adopt.
Thats plain wrong, the gloval market for cars, bicycles and what have you has a limited size. Every large company that's a market leader understands that.
If you do the math, then increasing your revenue by 2x doesn’t cost 2x the effort. But one additional percentage point of cost reduction might well cost 2x the effort.
Sadly, most employers I’ve known do not seem to understand this thing called math.
Instead, when something has a payoff of 3 years, executives get antsy in orgs that have a 2-year cycle on exec positions.
This point seems wrong to me, bound too much by requiring solutions to be done by a small team of engineers who already have a mandate to work on the problem.
With numbers like that Twitter could, profitably, hire dozens of engineers that do literally nothing else. Just tweak thread pool sizes all day, every day, for service after service. Even though it's a boring, manual, "noncomplex" thing, this type of work is clearly valuable and should have happened years ago.
Most likely Twitter's job ladder, promotion process, and hiring pipeline is highly incentivizing people to avoid such work even when it has clear impact. They are very much not alone in that regard.