We miss a lot of opportunities, and not taking them has lingering consequences.
Plenty of that going around.
Perf analysis was how I got through some of my most mundane college homework assignments instead of (or sometimes, in addition to) procrastinating.
Then during my formative years I saw the following script play out several times:
Management and sales have a problem. They say the customers are upset that the code is so slow (maybe the customer has expectations, or maybe a competitor or their old system went a lot faster). It's taking 300 units of time now and they think it should take 100. So the leads and the heavies wander off and come back with 20% easy, the next 10 harder, 6, 4, and then they declare that they've done all they can. So now we're around 180 units of time. That's all the customer gets. See this <flame chart precursor> chart? Everything is nice and flat. Our hands are tied now. Business is... not happy. They're glad they have something to show the customer, but it's really just a bone.
So between tasks I'd start poking around, because why not. A pinch here, an architectural tweak there, and maybe 25% just gruntwork, and I'd find another 40% out of the "we've done all we can" code, which is just a hair away from what the business asked for.
You can imagine that you invite a tremendous amount of well-deserved scrutiny on yourself doing stuff like this. But that's okay: the most important part of optimization is not the code speed but the bang for the buck. You can certainly interpret "Low hanging fruit" to fulfill that concern, in practice most people don't (for some, lack of imagination, for others, ego-protection). But once you consider everything, including, for instance, externalities on QA, or the misapprehensions of customers who expect every release to be faster... there are a lot of more complicated things to worry about than how to rearrange this block of code to improve branch prediction. Not touching it for 10% is just a heuristic.
When you are way over budget, every cost needs to go on the table, but your end goal is one of the most important pieces of information. That should go on the wall, in large block letters.
Look at a real budget. If you are deep in the red, it doesn't matter that your car is only 10% of how much you spend a month. What matters is that the car is 25% of your target. That is a ridiculous amount to spend on your car, even though your house is way more. Even if you spend an ungodly amount on dry cleaning or coffee, you're going to have to trade down for a cheaper car. You don't need charts to tell you this. In fact all the charts can do is convince you that nothing needs to be done. They allow you to bargain.
Therefore, I don't give a shit if you think that this function which accounts for "only 5%" of the current run time "is fine". It's 13% of our time budget on some mundane task that doesn't really make us money. It is not fine. You cannot justify 1/8th of our budget for this thing. It's gonna be fixed, and if you have no idea how, you'd better start thinking about it now, because we're gonna come back to it soon and if you don't have a solution then you might find that code is now someone else's responsibility.
But I said something about the 'fruit tree' analogy and I'm a full page of text in without a peep.
So the thing is that when you're thinking about big budgetary changes, it's more manageable to do it one subject matter at a time. Pick a 'ripe' area and glean it for everything it's worth. So I might pitch that I think I can get 30% improvement out of the edit/update code. Can I have this much time to do it? Okay. I might get half of that goal in the first couple weeks. Half of the rest in the next, but at the end of it, the last thing I touch is probably only going to get 3%. But 3% is 10% of what I promised, so I can justify it, and if they aren't complaining about the timeline, I may try to squeeze in some 1-2% things at the end in a way I can pull the plug if it looks too risky.
The important thing to note here is that it will never, ever be as cheap as it is right now to touch that 3% code. Everyone is already thinking about it, everyone knows to keep an eye on it, and new test plans are being developed, documentation and customer training plans are being updated. If I don't touch it now, that target of opportunity might never come again. That 3% plays out everywhere in the code, and I will never be able to get budget to fix any of them (I'll have to sneak them in on a refactor or leave them forever). A dozen of these is not that hard to end up with, and our code is 40% slower than it could be (or worst of all, than a competitor). If that's just the time for a button click, fine. If it's on our slowest operation and things time out, customers with larger data can't function, or they have to buy the absolute most expensive machines instead of slightly better than average? That's not fine. But everyone feels perfectly justified in never doing anything about it.
Next release, I can pitch for 25% in some other functional area, and another, and another. Last time I did this, it took over 20 months before I ran out of functional areas to work on. Every release for 'years' was faster than the previous. At the end, I knew that code better than anyone (an important point I omitted - focusing that intently on one section teaches people a hell of a lot about the code they otherwise would never know). I probably could have swung through for another 10% in every area, but by then I was ready for a new gig.
We all know that hill climbing algorithms are often naive and sometimes hilariously wrong. Nobody will disagree with you about this, until you start talking about prioritizing work, and then everyone vigorously defends their favorite hill climbing algorithm, from task management to performance tuning.
My preferred strategy for optimization more closely resembles how a fruit grower would pick fruit, and the term, “low-hanging fruit” galls me because you would go bankrupt using that strategy. And probably lose most of your trees to disease. It could be a quite good analogy, if the lessons taken from it weren’t just so childishly naive.