Those teams built up that kind of tacit knowledge like in this video about software texture mapping https://youtube.com/watch?v=xn76r0JxqNM
If I were to teach, which I kinda do as a leader of the team, I d just give a few shortcut: 99% of perf issues are stupid and discoverable by profiling (which nobody does somehow, the most experienced the least likely the dev will use a profiler and the most likely he ll have 50 wrong migration ideas to make everything better with a huge budget), once you compressed all leaves of code you can find on a hot path, look at why this hot path even exist, chances are it shouldnt even be called as often, if you did that and are still too slow, redesign, multi thread more, remove business need, expand the hardware.
You start also getting an instinct on scale: a troubled team behind by half an hour on a settlement process with an exchange, so big deal, told me they wanted to move the cloud to spin up hundreds of big server with a new scalable design that would take a year to develop with 0 skill in the team yet. How many operation do they need to do in the alloted time ? 30k, with a database on the back. Well, I havent looked yet, but this sounds to me like we can try something before the mega migration lol because 30k units of work is not much and databases tend to be misused (I once fixed a million statement sent by hibernate on a SOAP call, so...) but as you see I try not to give an explanation, it s just an instinct, because prejudice is what kills perf optimization: we'll have to profile each step.
One thing I try to teach is: never migrate to a new library while calling the slow code legacy: you'll end up either failing exactly the same way or fake-suceeding on a small sample and leave the actual failure to the next team who'll call your code legacy before moving to the next fancy fad framework of the year and start the cycle of disaster over again.
I've also removed a lot of 10% calls that improved performance by 20%. Profilers don't show the cost of your code to the CPU and Virtual Memory caches. And there are new or malloc calls whose full cost is paid by another thread due to the tempo of the allocations.
As you say, looking at the count column is a huge thing. But the biggest failure I've seen is one of imagination. So many people give up when the last tall tent pole has been addressed. Even if we've only hit a 3rd of our performance goal, they believe they've tried everything and we should move on. Nobody knows how to do perf analysis with a budget. For instance, the 'current users' count should not be entitled to 10% of the page load time.
Or better yet, how about just writing a series of blog posts or a newsletter?
Don't waste time on the setup, just register something on substack.com and I'm sure a bunch of folks here will subscribe. It sounds like a fascinating subject.
Oddly, my first specialty was performance optimization. A predisposition met a trouble project and I spent half my time finding ways to speed up code without making it offensive. Then later projects had problems but no mandate, so I got pushback. I got really good at crypto-optimization - changes that improve performance but look like something else (cleanup, bug fixes, feature requests), and found there’s a whole quadrant of code changes that answer both legibility and speed.
There is very little magic or intuition based about any of this. It could be in a book. If anyone knows of a book that covers this, I’d love to gift it to people, because as far as I’m aware I’m seen as a weird (semiretired) street preacher on this subject. But I’m just tweaking existing recipes as it were, adding common ingredients in uncommon ways.