News for hackers. Stuff that matters.
Re:Tuning Linux for Performance by dijit on Saturday October 14, 2023 @08:39AM (#37880012)
The issue with performance is that junior sysadmins and developers start flipping knobs thinking that the defaults are somehow holding them back.

The truth is usually there are tradeoffs and the defaults fit a broad general case.

If you want throughput there are tunables for that, if you want low latency then usually those are inversely correlated. Same for tuning for low data loss after failure and so on.

You have to spend time learning the tradeoffs, which sysadmins used to do- now nobody has time as they have been munged into one role at many places.

Re:Tuning Linux for Performance by eduction on Saturday October 14, 2023 @08:46AM (#37880057)
That may be, but Brendan Gregg is not a junior flipping switches, and a resource like this can help others do smart tuning.
Re:Tuning Linux for Performance by dijit on Saturday October 14, 2023 @08:48AM (#37880068)
I didnt expect anyone would read my comment that way.

No, Brendan Gregg is certainly no junior. The issue is that people take his advice as law and do not read further.

I, myself, remember flipping random switches because a (book) resource said that it would unlock performance.

I’m suggesting that taking the time to understand performance properly is ideal and I am attempting to urge people to properly invest the time.

Then I am lamenting the fact that we do not actually have time over because there’s so much smushing of responsibilities.

Re:Tuning Linux for Performance by eduction on Saturday October 14, 2023 @08:51AM (#37880095)
Ah fair enough, sorry if I misunderstood!
Re:Tuning Linux for Performance by traceroute66 on Saturday October 14, 2023 @09:39AM (#37880395)
> Brendan Gregg is not a junior flipping switches

That's because Brendan Gregg doesn't flip switches, he shouts at hard drives[1]. ;-)

[1] https://www.youtube.com/watch?v=tDacjrSCeq4

Re:Tuning Linux for Performance by pmcf on Saturday October 14, 2023 @12:18PM (#37881721)
I do the same thing to my kids to change their performance.
Re:Tuning Linux for Performance by cinntaile on Saturday October 14, 2023 @01:50PM (#37882595)
Does it work?
Re:Tuning Linux for Performance by bornfreddy on Saturday October 14, 2023 @02:00PM (#37882672)
Not GP, but no, it doesn't. It appears to work but then side effects get much worse. Taking the time to understand and to tweak the right settings, I mean, explain, is much better in the long run.
Re:Tuning Linux for Performance by efortis on Saturday October 14, 2023 @09:36AM (#37880378)
I wouldn't say that that's a performance tuning, seniority, or even smart people problem per se. I've seen smart people blindy tuning knobs. Similarly, switching to an expensive O(1) algorithm for a tiny space. It beats me, but I think the problems are closer to lazyness or trying to discredit something.
Re:Tuning Linux for Performance by karmakaze on Saturday October 14, 2023 @10:12AM (#37880609)
This broad case is much broader than the typical servers that Linux is often used for. A simple example is file access times on a (database) server. These are largely unnecessary. It's even rare for a desktop user to actually look at these.

In the past I've been a sr dev (but a jr sysadmin) and was tasked with improving the performance of an upgraded database server. The problem turned out to be with NUMA on the larger server which a combination of reading and semi-random config fiddling of both Linux and MySQL parameters (plus a bit of BIOS/CMOS tweaking) brought up to expected levels.

There's often no better way than learning on the job as there is so much to know that you can't simply learn them up front for when you will need it. What we can do is learn what there is to know and remember to look into those if it seems relevant. I mean everyone who's well experienced now probably started config twiddling somewhere to get there.

Re:Tuning Linux for Performance by unmole on Saturday October 14, 2023 @12:37PM (#37881929)
> A simple example is file access times on a (database) server.

This isn't a issue if you use `relatime` which is the default.

Re:Tuning Linux for Performance by pflanze on Saturday October 14, 2023 @01:14PM (#37882297)
To give context for those who don't know: relatime is a relatively (I'm a dinosaur) newer introduction into the Linux kernel; mounting partitions noatime used to be the only alternative to the default which was updating the atime on every access, thus fiddling with that setting used to be important. Not any more.

BTW I used to continue to mount everything noatime anyway, since having the atime field be set upon file creation and not anymore afterwards was a way to get file creation time, and I found that more useful than the access time. This isn't necessary either anymore since the introduction of an actual file creation time.

Re:Tuning Linux for Performance by js2 on Saturday October 14, 2023 @12:54PM (#37882108)
There's a right and a wrong way to learn though. The wrong way, which is the one I most often see is: smash keys until it appears to work w/o every bothering to try to understand the deeper problem or why some particular combination of key smashing appears to work. This becomes cargo culting over time. Stack Overflow and ChatGPT make this oh so much worse. Often the change then get committed with a useless message such as "make thing work" w/o any explanation of what was broken, what the change fixed, and why or how. It becomes instant tech debt.

The right way is to follow the scientific method. Collect data. Make a hypothesis with a plausible mechanism of action. Test the hypothesis. Arrive at a solution. Record how you arrived at your new found knowledge so that those who follow you understand why you made the change. The person who follows you is your future self as often as not.

Re:Tuning Linux for Performance by nimbius on Saturday October 14, 2023 @10:37AM (#37880801)
though i dont know much about it, i run linux and most of the documents i hear about tuning edge cases are now either avoided or boilerplated with a strong word of caution. For example: 10Gb ethernet used to require a litany of sysctl.conf chicanery to even approach half the line speed. not the case anymore, and most of the old kernel 2.4 optimisations are either nonsensical in 2023 or actively worsen the performance of the interface.
Re:Tuning Linux for Performance by tyingq on Saturday October 14, 2023 @11:01AM (#37880980)
> The truth is usually there are tradeoffs and the defaults fit a broad general case.

Part of the problem is that reasonable defaults for performance is a somewhat new phenomenon. It used to be that the defaults for Linux kernel settings, Apache, MySQL, etc... were terrible for production use.

So there's a lot of history that "you have to change them" burned into people minds, documents, etc.

Re:Tuning Linux for Performance by vbezhenar on Saturday October 14, 2023 @11:59AM (#37881546)
The truth is that often nobody thinks too much of defaults, unless they are horribly wrong. So there are two good things about defaults: they're probably not horribly wrong and they don't require any additional work.

Some defaults are just historical curiosities, some defaults were configured 20 years ago and nobody took the crusade to update them, some might be bad, but changing them would break too much stuff in the wild.

Now I'm not suggesting that everyone should change everything. I almost never change defaults, myself. But I just don't agree than defaults are good. They're probably not bad, and that's about it.

Re:Tuning Linux for Performance by jeffbee on Saturday October 14, 2023 @12:13PM (#37881659)
> defaults fit a broad general case.

Yeah, they fit the general case of a MIPS R4400 with a 1mbps network adapter, a situation that nobody faces today. I think the most glaring example is `rmem_max` which is never sufficient to support a coast-to-coast 1gbps flow, and every individual Linux user in history has needed to independently discover this stupid sysctl.

Re:Tuning Linux for Performance by ranting-moth on Saturday October 14, 2023 @01:10PM (#37882263)
The problem you're explaining can be generalized:

"Junior X starts flipping knobs thinking Y is somehow holding him back"

This is sadly how many of us have to learn. If you don't have a mentor or someone to check your work, it's a way to learn.

Experience will tell you not to make bad decision. You get experience by making the wrong decisions.

Re:Tuning Linux for Performance by klysm on Saturday October 14, 2023 @01:13PM (#37882295)
Most of the time I senior folks say benchmarking is the only way to know. So I'm not sure that this piece of advice is good.