No, Brendan Gregg is certainly no junior. The issue is that people take his advice as law and do not read further.
I, myself, remember flipping random switches because a (book) resource said that it would unlock performance.
I’m suggesting that taking the time to understand performance properly is ideal and I am attempting to urge people to properly invest the time.
Then I am lamenting the fact that we do not actually have time over because there’s so much smushing of responsibilities.
That's because Brendan Gregg doesn't flip switches, he shouts at hard drives[1]. ;-)
In the past I've been a sr dev (but a jr sysadmin) and was tasked with improving the performance of an upgraded database server. The problem turned out to be with NUMA on the larger server which a combination of reading and semi-random config fiddling of both Linux and MySQL parameters (plus a bit of BIOS/CMOS tweaking) brought up to expected levels.
There's often no better way than learning on the job as there is so much to know that you can't simply learn them up front for when you will need it. What we can do is learn what there is to know and remember to look into those if it seems relevant. I mean everyone who's well experienced now probably started config twiddling somewhere to get there.
This isn't a issue if you use `relatime` which is the default.
BTW I used to continue to mount everything noatime anyway, since having the atime field be set upon file creation and not anymore afterwards was a way to get file creation time, and I found that more useful than the access time. This isn't necessary either anymore since the introduction of an actual file creation time.
The right way is to follow the scientific method. Collect data. Make a hypothesis with a plausible mechanism of action. Test the hypothesis. Arrive at a solution. Record how you arrived at your new found knowledge so that those who follow you understand why you made the change. The person who follows you is your future self as often as not.
Part of the problem is that reasonable defaults for performance is a somewhat new phenomenon. It used to be that the defaults for Linux kernel settings, Apache, MySQL, etc... were terrible for production use.
So there's a lot of history that "you have to change them" burned into people minds, documents, etc.
Some defaults are just historical curiosities, some defaults were configured 20 years ago and nobody took the crusade to update them, some might be bad, but changing them would break too much stuff in the wild.
Now I'm not suggesting that everyone should change everything. I almost never change defaults, myself. But I just don't agree than defaults are good. They're probably not bad, and that's about it.
Yeah, they fit the general case of a MIPS R4400 with a 1mbps network adapter, a situation that nobody faces today. I think the most glaring example is `rmem_max` which is never sufficient to support a coast-to-coast 1gbps flow, and every individual Linux user in history has needed to independently discover this stupid sysctl.
"Junior X starts flipping knobs thinking Y is somehow holding him back"
This is sadly how many of us have to learn. If you don't have a mentor or someone to check your work, it's a way to learn.
Experience will tell you not to make bad decision. You get experience by making the wrong decisions.
The truth is usually there are tradeoffs and the defaults fit a broad general case.
If you want throughput there are tunables for that, if you want low latency then usually those are inversely correlated. Same for tuning for low data loss after failure and so on.
You have to spend time learning the tradeoffs, which sysadmins used to do- now nobody has time as they have been munged into one role at many places.