Both have similar problems: if the business goals don’t align with reducing latency or improved security posture then your detailed analysis will get parked in a ticket queue and promptly forgotten. Your OKRs will be missed. You will also be fighting the tide against new features developed sloppily- as latency will increase and then investment in your team seems like it’s not working to the executives. Fortunately we had a supportive directive but being siloed into an infra org didn’t give us the juice to influence product roadmaps.
We built a lab for testing against devices, did a bunch of work to isolate noise, integrated automated regression testing into CI to combat this. It still didn’t move the needle frankly, as the teams building products were not held accountable for the regressions (and don’t forget that it’s likely the right move for the greater business health!).
Netflix is unique in that latency is very core to their product.
The principle to do whatever you love is to "ask for forgivness not permission". Feel free to understand why you're solving what you're solving in order to challenge it and do it right.
But unfortunate it was a totally unoptimized thing so only quick low hanging fruits.
Like io bound but they didn't though that component is the issue as it was not CPU bound, bad SQL queries and unoptimized caches.
Nothing big or difficult to analyse :-(
I learned about one person who is only doing java Performance analysis and he said it took a while to be known in the industry good enough that he can do it fulltime.
And he has to travel to allot of customers.
https://www.brendangregg.com/blog/2021-05-29/moving-to-austr...
I even hoped joining Netflix in the future to say hi to him :)
Doing performance analysis feels like the final thing to fully understand the whole stack and I always was curios on how things work.
Let's see were he will go. Perhaps it's easier to follow but I doubt it.
There are companies I'm following for a while because certain positions are just seldom pop up.
When I joined sap I heard the term headcount for the first time but makes it more understandable.