News for hackers. Stuff that matters.
Re:Linux with “memory folios”: a 7% performance boost when c... by marcodiego on Monday June 14, 2021 @10:04PM (#27510539)
Now, consider that there's work going on to enable Linux to be compiled with profile guided optimization with clang[0], the DAMON patchset that enables proactive reclamation with "32% memory saving with only 1.91% runtime overhead"[1] and performance improvements achievable with futex2 system call[2].

Linux future seems bright with regard to performance.

[0] https://lkml.org/lkml/2021/1/11/98

[1] https://lore.kernel.org/lkml/20210608115254.11930-1-sj38.par...

[2] https://lkml.org/lkml/2021/4/27/1208

Re:Linux with “memory folios”: a 7% performance boost when c... by londons_explore on Monday June 14, 2021 @11:18PM (#27510891)
Linux needs a standard set of benchmarks that are vaguely representative of things users use Linux for. Phone, laptop and a few server use cases.

It then needs someone to do a big parameter tuning to select optimal settings.

Too many decent algorithms don't make it into the kernel because there are too many tunables, and the ones that do typically arent well tuned for anyone's use case.

Even big projects like Ubuntu typically don't change many tunables in the kernel.

Re:Linux with “memory folios”: a 7% performance boost when c... by viraptor on Tuesday June 15, 2021 @01:32AM (#27511663)
> Linux needs a standard set of benchmarks that are vaguely representative of things users use Linux for.

Have you seen phoronix framework + benchmarks? They use common tools, but did some good work on making the tests repeatable and accessible to anyone else.

https://github.com/phoronix-test-suite/phoronix-test-suite/

Re:Linux with “memory folios”: a 7% performance boost when c... by fgeiger on Tuesday June 15, 2021 @03:55AM (#27512483)
Linaro is also doing a lot in that space (at least for ARM): https://www.linaro.org/os-build-and-test/
Re:Linux with “memory folios”: a 7% performance boost when c... by beebeepka on Tuesday June 15, 2021 @04:00AM (#27512514)
Just minor nitpick. Phoronix is one guy
Re:Linux with “memory folios”: a 7% performance boost when c... by viraptor on Tuesday June 15, 2021 @08:59AM (#27514686)
Correct, but I like British grammar, where companies are plural :-)
Re:Linux with “memory folios”: a 7% performance boost when c... by eptcyka on Tuesday June 15, 2021 @05:04AM (#27512847)
The test suite is open source, I'm sure Michael would appreciate some help :)
Re:Linux with “memory folios”: a 7% performance boost when c... by stewartmcgown on Tuesday June 15, 2021 @04:54AM (#27512781)
they can be singular
[dead]
Re:Linux with “memory folios”: a 7% performance boost when c... by franciscop on Tuesday June 15, 2021 @05:32AM (#27512981)
Only if "They" is their given name and not a pronoun. If used as a pronoun, you can say "they are tall" in singular:

https://apastyle.apa.org/style-grammar-guidelines/grammar/si...).

Re:Linux with “memory folios”: a 7% performance boost when c... by bradleyjg on Tuesday June 15, 2021 @05:50AM (#27513069)
Seems like a violation of the strict aliasing rule.
Re:Linux with “memory folios”: a 7% performance boost when c... by andrepd on Tuesday June 15, 2021 @06:29AM (#27513304)
How so?
Re:Linux with “memory folios”: a 7% performance boost when c... by CyberShadow on Tuesday June 15, 2021 @07:50AM (#27513922)
Because you use a "multiple" of pointers to refer to a single entity. (It's a C joke.)
Re:Linux with “memory folios”: a 7% performance boost when c... by mdp2021 on Tuesday June 15, 2021 @06:04AM (#27513161)
That page you link to uses very ideological content - not philological as one could expect.

Not all will agree with those guidelines, they are far from universally acceptable.

Re:Linux with “memory folios”: a 7% performance boost when c... by PavleMiha on Tuesday June 15, 2021 @07:24AM (#27513701)
Think about it this way, we're playing Guess Who, and you haven't asked about gender yet. You ask "Do they have dark hair?". It's obviously a question about a singular person, and that's the correct way to structure that question in English.
Re:Linux with “memory folios”: a 7% performance boost when c... by mdp2020 on Thursday June 24, 2021 @05:46AM (#27615589)
Certainly. But I referred to the presented guidelines, collectively - not to "singular they". I wrote and intended that _those linked guidelines_ are "not universal".
Re:Linux with “memory folios”: a 7% performance boost when c... by vidarh on Tuesday June 15, 2021 @08:21AM (#27514202)
Singular they in English usage dates back 600-700 years. People can disagree with those guidelines all they want - they will have to deal with the fact that people choose to use singular they and they can get worked up over it or get used to it.
Re:Linux with “memory folios”: a 7% performance boost when c... by bradleyjg on Tuesday June 15, 2021 @08:58AM (#27514669)
Do you feel the same way about generic he? It too has a long history.
Re:Linux with “memory folios”: a 7% performance boost when c... by roelschroeven on Tuesday June 15, 2021 @09:36AM (#27515314)
Yes, generic he can be used in the singular as well.
Re:Linux with “memory folios”: a 7% performance boost when c... by vidarh on Tuesday June 15, 2021 @12:08PM (#27517472)
It is clearly valid. I personally don't care if others use "he" as a generic, but seeing at some people don't feel included when it is used, I increasingly use "they". Not least when I don't know someones preference, because I consider the most inclusive option to be the most polite.
Re:Linux with “memory folios”: a 7% performance boost when c... by bradleyjg on Tuesday June 15, 2021 @12:17PM (#27517591)
I don’t feel included by they.
Re:Linux with “memory folios”: a 7% performance boost when c... by vidarh on Tuesday June 15, 2021 @12:19PM (#27517624)
You objectively are, and in the choice of who to care about that means I have absolutely no interest in catering to people upset that others are included.
Re:Linux with “memory folios”: a 7% performance boost when c... by bradleyjg on Tuesday June 15, 2021 @01:16PM (#27518427)
There’s no difference objectively between generic he and generic they. Both cover everyone. You just prefer one to the other. That’s fine, so do I.
Re:Linux with “memory folios”: a 7% performance boost when c... by vidarh on Tuesday June 15, 2021 @01:30PM (#27518638)
There is an objective difference between generic he and generic they: In many contexts you can not tell whether the "he" was intended to be generic or not.
Re:Linux with “memory folios”: a 7% performance boost when c... by mdp2020 on Thursday June 24, 2021 @05:54AM (#27615647)
I did not write anything about "singular they". I wrote against the legitimacy of those linked guidelines, which accidentally include "singular they".

The poster I replied to stated (I remember) that some use of language is made legitimate by some group laying out some guidelines. That some pretty random group («not philologists») lays out guidelines is maybe an "affiliation pass" for their group, but not universally valid.

You need grounds, good grounds. Check those guidelines...

Re:Linux with “memory folios”: a 7% performance boost when c... by andrepd on Tuesday June 15, 2021 @06:30AM (#27513319)
How so philological? Language changes all the time due to practice. People don't need to justify their speech with philological arguments.
Re:Linux with “memory folios”: a 7% performance boost when c... by moomin on Tuesday June 15, 2021 @07:54AM (#27513966)
Yes, although in the case of singular they, its usage goes back to Chaucer, so it’s more established than most practices in modern English.
Re:Linux with “memory folios”: a 7% performance boost when c... by mdp2020 on Thursday June 24, 2021 @05:47AM (#27615600)
I believe they do. Practice must have a reasonable reference. People are free to contribute to the language - but with some competence.
Re:Linux with “memory folios”: a 7% performance boost when c... by majormajor on Tuesday June 15, 2021 @10:56AM (#27516420)
Where's the ideological content?

Some people: "let's use, of already-established pronouns, the one that doesn't make any assumptions about people" You: "no, fuck you, other people's interpretation of what I say is strictly their problem, I'm gonna stick with 'he' no matter what other people feel or request, because I can"

The only ideology I can see here is the absolutist sort of "nobody can even suggest to me that I change a thing" or "I can't possibly cause offense if I don't mean to, so I don't need to think about word usage."

Re:Linux with “memory folios”: a 7% performance boost when c... by bradleyjg on Tuesday June 15, 2021 @11:16AM (#27516726)
I don’t understand how it doesn’t make assumptions about people. In the case of a generic antecedent, sure that’s fine and well established historically. But in this case there’s a known antecedent, Michael Larabel.

I don’t know what Michael’s preferred pronouns are, but isn’t the original poster in this chain assuming it’s they/them and aren’t they more likely, statistically speaking, to be he/him?

Why does it somehow not count as misgendering when you they/them someone that prefers he/him or she/her?

Re:Linux with “memory folios”: a 7% performance boost when c... by vidarh on Tuesday June 15, 2021 @12:25PM (#27517719)
> Why does it somehow not count as misgendering when you they/them someone that prefers he/him or she/her?

Because one appears to make an assumption about gender, whether or not you intended it to be taken as a generic, while the other objectively is generic.

As far as options go, they/them minimises assumptions. Yes, some people might still take offence, but given that most of the people who take offence at that takes offence because they oppose inclusiveness of others, I'm perfectly fine with not respecting their choice in the matter.

Re:Linux with “memory folios”: a 7% performance boost when c... by bradleyjg on Tuesday June 15, 2021 @01:19PM (#27518468)
but given that most of the people who take offence at that takes offence because they oppose inclusiveness of others

You have some evidence of this or just the unshakable certainty of the self righteous?

Re:Linux with “memory folios”: a 7% performance boost when c... by vidarh on Tuesday June 15, 2021 @01:29PM (#27518627)
I go by experience. There may be exceptions, but I've yet to see one, so I don't particularly care if there's a large pool of exceptions outside the horizon of my personal experience.

If you are an exception and have a good reason for taking offence that doesn't involve excluding others, do tell.

Re:Linux with “memory folios”: a 7% performance boost when c... by andrepd on Tuesday June 15, 2021 @06:20AM (#27513264)
No, but "They are tall" is, which is gramatically plural but semantically singular.
Re:Linux with “memory folios”: a 7% performance boost when c... by jraph on Tuesday June 15, 2021 @07:42AM (#27513854)
Is it grammatically plural? In "You are tall", "you" can be singular, and then so should "are" be, if I am not mistaken.

(I'm not a native English speaker)

Re:Linux with “memory folios”: a 7% performance boost when c... by umanwizard on Tuesday June 15, 2021 @07:54AM (#27513958)
“You” was exclusively plural (singular equivalent was “thou”), but its meaning shifted to be either singular or plural a few hundred years ago. So “are” was indeed a plural-only conjugation until that shift happened.

Now “you” is shifting again: in formal English it is still singular or plural, but in colloquial American English — at least where I’m from — it can only be singular. The plural form is “you guys” or “y’all”, depending on dialect. (The “guys” in this pronoun shouldn’t be confused with the noun “guys”, which is usually only used for boys and men; “you guys” is used to refer to groups of any gender).

Re:Linux with “memory folios”: a 7% performance boost when c... by andrepd on Tuesday June 15, 2021 @09:08AM (#27514816)
Exactly. There's loads of examples of such shifts in other languages, for example in Portuguese the formal you is conjugated in grammatical third person despite being semantically second person, arising from people being formally addressed in the third person ("Does the Right Honourable gentleman agree that...").
Re:Linux with “memory folios”: a 7% performance boost when c... by umanwizard on Tuesday June 15, 2021 @09:42AM (#27515409)
Same is true in Spanish, where "usted" ultimately derives from "vuestra merced".

For a completely different example, "on" in formal French means something like "one" in English: a pronoun for a general, unspecified person; as such, it takes singular conjugations. However, in colloquial speech, it means "we" and has almost completely replaced "nous" as a subject pronoun.

Thus: "il est" (he is); "on est" (we are, colloquial language); "nous sommes" (we are, standard language)

Re:Linux with “memory folios”: a 7% performance boost when c... by scatters on Tuesday June 15, 2021 @05:46AM (#27513054)
Is "You art tall." a grammatically valid sentence? Obviously not; don't be silly.
Re:Linux with “memory folios”: a 7% performance boost when c... by acer4666 on Tuesday June 15, 2021 @04:45AM (#27512742)
Without knowledge of preferred pronouns, "they" seems appropriate.
Re:Linux with “memory folios”: a 7% performance boost when c... by fgeiger on Tuesday June 15, 2021 @05:23AM (#27512933)
As I read it, the point was not about the wrong pronoun but about the fact that the suite is developed by one person and not a large team and that the burden of benchmarking Linux performance should maybe not rest on one person alone.
Re:Linux with “memory folios”: a 7% performance boost when c... by rat9988 on Tuesday June 15, 2021 @05:42AM (#27513034)
You missed his point. There was no wrong pronoun, this is what he is claiming.
Re:Linux with “memory folios”: a 7% performance boost when c... by UbrtrbNchDneRle on Tuesday June 15, 2021 @04:53AM (#27512776)
viraptor 1 : 0 the west.
Re:Linux with “memory folios”: a 7% performance boost when c... by leadingthenet on Tuesday June 15, 2021 @06:01AM (#27513142)
...and people like to pretend this kind of nonsense hasn't penetrated into every social space.
Re:Linux with “memory folios”: a 7% performance boost when c... by jraph on Tuesday June 15, 2021 @07:47AM (#27513897)
It unfortunately still hasn't penetrated into the most old school ones, apparently. Whatever you are talking about anyway.
Re:Linux with “memory folios”: a 7% performance boost when c... by 5e92cb50239222b on Tuesday June 15, 2021 @08:27AM (#27514278)
If by "old school" you mean "everywhere outside of Silicon Valley", then maybe. Coming from a pretty conservative society, I read these sorts of discussions with mild amusement. I may have formed a mistaken impression of the US society, but it seems to me that it tends to bounce from one extreme of political or social views to another.
Re:Linux with “memory folios”: a 7% performance boost when c... by jraph on Tuesday June 15, 2021 @08:39AM (#27514391)
I would not know, I live in Europe ;-) Extrema do exist here though.
Re:Linux with “memory folios”: a 7% performance boost when c... by IntelMiner on Tuesday June 15, 2021 @07:00AM (#27513524)
A single data point does not an argument make
Re:Linux with “memory folios”: a 7% performance boost when c... by MayeulC on Tuesday June 15, 2021 @11:53AM (#27517273)
But Michael tends to use "we" in his writings.
Re:Linux with “memory folios”: a 7% performance boost when c... by mhh__ on Tuesday June 15, 2021 @12:38AM (#27511352)
One of the things that struck me recently is that working out what a computer is actually spending it's time doing is now almost a Master's thesis to do properly.
Re:Linux with “memory folios”: a 7% performance boost when c... by adrianN on Tuesday June 15, 2021 @12:42AM (#27511372)
Depending on the level of analysis you can even get a PhD. See for example https://core.ac.uk/download/pdf/18462805.pdf
Re:Linux with “memory folios”: a 7% performance boost when c... by mhh__ on Tuesday June 15, 2021 @12:46AM (#27511393)
Not surprising at all, sadly.
Re:Linux with “memory folios”: a 7% performance boost when c... by bayindirh on Tuesday June 15, 2021 @02:57AM (#27512173)
I've found it the hard way when I was trying to profile my own code.

If you're using a few optimized libraries and designed your code for high-speed parallel execution, the compiled thing becomes almost impossible to trace in a practical manner.

Re:Linux with “memory folios”: a 7% performance boost when c... by rwmj on Tuesday June 15, 2021 @06:01AM (#27513138)
Didn't perf work? What difficulties did you encounter?
Re:Linux with “memory folios”: a 7% performance boost when c... by bayindirh on Tuesday June 15, 2021 @07:26AM (#27513717)
Used perf record for getting CPU efficiency numbers, and it worked very well, however to be able to reliably trace the code, I had to compile it in debug mode with no optimizations, and run with a single thread. That combination increased the execution time considerably.

Some of the Matrix and solver code coming from Eigen is heavily optimized for SIMD operations and using it with -O0 is just painful.

Moreover, I had to verify its memory sanity and used Valgrind for that. Using a full size problem meant it had to churn for days to finish execution.

Having deadlines doesn't always help in this stuff.

Re:Linux with “memory folios”: a 7% performance boost when c... by mhh__ on Tuesday June 15, 2021 @11:21AM (#27516776)
I found perf pretty crummy compared to vTune and even vTune is not fantastic at scale.
Re:Linux with “memory folios”: a 7% performance boost when c... by bayindirh on Tuesday June 15, 2021 @11:46AM (#27517153)
I think perf and valgrind is a fantastic combination. While valgrind has a big overhead, it can make a lot of things visible without instrumenting the code.

I also think Perf is phenomenal for its scope. It's showing performance metrics of the code without any processor dependency and instrumentation.

Re:Linux with “memory folios”: a 7% performance boost when c... by Bayart on Tuesday June 15, 2021 @05:15AM (#27512895)
On the positive side of things, the notion that performance is multivarious and means different things to different contexts has really sunk in with the general tech demographic and we don't see much of the "one big number that sums it all" type of stuff. Think the GHz race or FPS measuring contests.
Re:Linux with “memory folios”: a 7% performance boost when c... by roflc0ptic on Tuesday June 15, 2021 @08:27AM (#27514270)
Both with the GHz race and FPS measuring contests, we've gotten to the point where there's substantially diminishing returns - GHz has stopped growing in the same way, and reaching the limits of human perception with FPS is generally readily achievable, and so it loses interest as a contest.

I suspect your broader point stands - certainly "performance is multivarious and means different things in different contexts" has sunk in for me. But I suspect the actual difficulty of reasoning about performance deters most people from doing it beyond big O notation. The fact that I'm aware that performance is complicated doesn't drive me to understand the complexity, I mostly just say "good enough" and move on.

Re:Linux with “memory folios”: a 7% performance boost when c... by nitrogen on Tuesday June 15, 2021 @02:11PM (#27519269)
reaching the limits of human perception with FPS

Nothing can hit 200fps at 4K, which you'd need to reach low enough latencies to be imperceptible.

Re:Linux with “memory folios”: a 7% performance boost when c... by voldacar on Monday June 14, 2021 @11:26PM (#27510935)
I wonder if someday you could make a differentiable kernel and do gradient descent in kernel-parameter-space
Re:Linux with “memory folios”: a 7% performance boost when c... by Daishiman on Tuesday June 15, 2021 @01:45AM (#27511734)
I believe Google already does this for setting their server's tunables.
Re:Linux with “memory folios”: a 7% performance boost when c... by eru on Tuesday June 15, 2021 @03:44AM (#27512423)
At Google's scale, you don't even need it to be differentiable. You can just do simulated annealing or so.
Re:Linux with “memory folios”: a 7% performance boost when c... by guenthert on Tuesday June 15, 2021 @02:58PM (#27519912)
Right, problem solved. Now we just need to identify the few representative use cases. Is it people using a phone, as, er, phone with actual voice communication? or video conferencing? Or those who play games casually or not-quite-so-casually? Or video streaming? Or just for texting and would rather prioritize security and battery runtime?

And that's just the phone. I could come up with many more questions regarding desktop and server usage as I'm more familiar with those.

I'm afraid, there is no one optimal set of configuration options. Not even two or three.

Re:Linux with “memory folios”: a 7% performance boost when c... by api on Tuesday June 15, 2021 @10:30AM (#27516034)
A few server use cases? It's used for like 99% of all server use cases.
Re:Linux with “memory folios”: a 7% performance boost when c... by diegocg on Tuesday June 15, 2021 @02:43AM (#27512078)
There are already plenty of benchmarks
Re:Linux with “memory folios”: a 7% performance boost when c... by astrange on Monday June 14, 2021 @11:39PM (#27511017)
Tuning is highly overrated in software performance, eg compiler optimizations don't help that much and searching their params helps even less. I think it comes from people wishing they could change their software without actually needing to learn how to change it.
Re:Linux with “memory folios”: a 7% performance boost when c... by dspillett on Tuesday June 15, 2021 @03:34AM (#27512369)
It is overrated when you are tuning the wrong thing, but if it is the bottleneck in a significant process for you use case then it is often very significant especially at scale. For someone running a compute cluster for their own needs or as a cloud service a couple of percent improvement like this represents a massive gain in throughput or a saving in the need for extra kit (or power costs - there may even be a small environmental benefit).

Of course on the scale of you or I, such optimisations may be little more than a curiosity most of the time. A process that normally takes 100 minutes (a video transcode, perhaps) now taking 95 followed by the machine being idle while it waits for us to look back and see the result, is not really benefiting from the benefit.

> without actually needing to learn how to change it

This is certainly a problem sometimes, but not an argument for tuning bring overrated in general. It usually comes down to tuning the wrong thing, like someone playing with mysql engine & kernel IO parameters to eek out a fraction of a of % bonus when they could improve index structures, or fix queries with no sargable predicates, or both in unison, to get benefits measured in orders of magnitude.

It can be overrated in the case of average gamers tweaking their hardware to get a few extra FPS on top of the many tens they already get, but again this not being worth the time for some, even many, doesn't mean it can make a huge difference to a pro-gamer or someone using old kit where "a few extra" is a relatively large gain.

Re:Linux with “memory folios”: a 7% performance boost when c... by jeffbee on Monday June 14, 2021 @11:47PM (#27511054)
That's quite a bad take. You can bank on PGO+LTO of a C++ server being good for 20% throughput easily, and not compiling for k8-generic when you have a modern machine also makes a big difference. I would agree that just flipping compiler flags might not get you very far, but there are key inputs that you shouldn't leave on the table.
Re:Linux with “memory folios”: a 7% performance boost when c... by astrange on Tuesday June 15, 2021 @12:21AM (#27511261)
By "not that much" I mean orders of magnitude. 20% is reasonable, especially for C++ that needs a lot of inlining.

But a more modern language would benefit from defining some optimizations like inlining as mandatory, the same way tail calls are (and of course GCC has this.) Then there isn't a chance of deploying a build with extra-slow behavior.

Re:Linux with “memory folios”: a 7% performance boost when c... by josephg on Tuesday June 15, 2021 @12:42AM (#27511374)
A performance improvement of 20% would be a massive win for something as large and widely used and as the linux kernel.

That would average out into a few percentage points of performance improvement in every application across the globe which runs on linux / android. Merging a patch like that in linux would be like quietly reaching into every device across the planet and giving them a small, free CPU improvement and drop in power usage. Not game changing for any individual user, but huge in aggregate.

Re:Linux with “memory folios”: a 7% performance boost when c... by arp242 on Tuesday June 15, 2021 @01:30AM (#27511654)
And if you run a lot of servers it means you need fewer servers. These kind of changes can literally save millions.
Re:Linux with “memory folios”: a 7% performance boost when c... by jeffbee on Tuesday June 15, 2021 @01:31AM (#27511661)
Which is why the organizations which stand to save millions by building their kernels with PGO already do it.
Re:Linux with “memory folios”: a 7% performance boost when c... by josephg on Tuesday June 15, 2021 @04:28AM (#27512654)
Some do. I bet lots of companies miss out on massive cost savings like this simply because nobody at the company has the skills and access to improve things.

I heard a story - years ago Google hired an engineer who happened to be an expert from a previous life in video codecs. He wasn’t working on YouTube at Google, but out of interest he pulled up the YouTube source code to see what it did. They were just using the defaults for some of the encoding parameters. He tweaked a few of the encoding parameters and in doing so saved Google millions of dollars per year in compute/storage/network traffic. A few hours of his work was probably more beneficial for Google than years spent in his primary role.

And if tuning opportunities like this abound at Google, you know they’re everywhere. It’d be much better if Linux distributions simply shipped kernels which are already compiled with PGO, with a reasonable profile based on normalish use.

Re:Linux with “memory folios”: a 7% performance boost when c... by astrange on Tuesday June 15, 2021 @02:59PM (#27519924)
That could've happened because of pointless detuning - x264 comes with good defaults for everything, but then ffmpeg on top of it used to set everything to "off" no matter what it was, and they were probably using the ffmpeg settings.

The correct answer would've been for ffmpeg to not ship that way.

Anyway, I am an expert on video codecs just like that guy is, and I said what I said ;)

Re:Linux with “memory folios”: a 7% performance boost when c... by arp242 on Tuesday June 15, 2021 @08:28AM (#27514291)
Probably, but there are many smaller outfits that probably don't because they have a server fleet of "only" a few dozen servers or so. The cost savings for those won't be in the millions, but having to purchase ~5% fewer servers is still a pretty good win.
Re:Linux with “memory folios”: a 7% performance boost when c... by jeffbee on Tuesday June 15, 2021 @09:56AM (#27515587)
Most smaller orgs are wasting resources on poor utilization. Tighter code just gets those orgs even worse utilization.
Re:Linux with “memory folios”: a 7% performance boost when c... by yokaze on Tuesday June 15, 2021 @01:38AM (#27511696)
Counterexamples:

Running your DB with the default parameters.

Not running Linux kernel parameters depending on your workload. (+10Gbit NICs, lots of connections...)

There are reasons, why those knobs are there, as there are always trade-off, and having them automatically adjust is very hard.

But maybe that was what you're referring to.

Re:Linux with “memory folios”: a 7% performance boost when c... by cycomanic on Tuesday June 15, 2021 @02:52AM (#27512146)
That highly depends on your problem. I have some scientific code that gets more than a factor 10 improvement on runtime just by turning on some compiler flags (mainly -O3 and -march, but some others as well).
Re:Linux with “memory folios”: a 7% performance boost when c... by oriolid on Tuesday June 15, 2021 @03:54AM (#27512481)
And to illustrate the point more, I have found that -Os often produces faster code than -O3. Not by factor 10, but still clearly measurable.
Re:Linux with “memory folios”: a 7% performance boost when c... by jeffbee on Tuesday June 15, 2021 @11:18AM (#27516747)
Getting better performance with -Os over -O is some kind of edge case and people shouldn't generally expect that. -Os has disastrous consequences for C++ algorithms because it refuses to inline functors, so while you may rightly believe that a C++ std::sort is slightly faster than a C qsort, due to superior opportunities to optimize the C++ code, with -Os you'll find that std::sort is an order of magnitude slower. Definitely pays to check the result with a full-scale benchmark.
Re:Linux with “memory folios”: a 7% performance boost when c... by oriolid on Tuesday June 15, 2021 @12:50PM (#27518050)
Interesting claim. I had to try this, and with the Xcode version that I have installed, std::sort with a simple lambda as comparison function gives about the same result with -O3 and -Os. I would not call this a disastrous, but of course opinions differ. Interestingly, qsort is significantly faster with -O3 than with -Os but of course nowhere near std::sort.
Re:Linux with “memory folios”: a 7% performance boost when c... by jeffbee on Tuesday June 15, 2021 @01:26PM (#27518579)
Results will vary for small vs. large programs. I've seen catastrophic space optimizations that out-of-lined very small methods like std::vector::at, because the call was one or two bytes smaller than the inline. Lambdas are inlined even with Os because they don't have names or multiple callers and can't be made smaller by out-of-lining. A functor class, or any function with multiple call sites could trigger the problems with Os.
Re:Linux with “memory folios”: a 7% performance boost when c... by oriolid on Tuesday June 15, 2021 @01:52PM (#27518952)
Ok, I'm not continuing with this. Just note that I didn't write that -Os would be always or even usually faster. I could try to come up with an example where -O3 produces a huge loop preamble for loop that's iterated once or just generates enough cache misses to be overall slower, but I don't care enough.
Re:Linux with “memory folios”: a 7% performance boost when c... by bayindirh on Tuesday June 15, 2021 @03:35AM (#27512372)
I also have a similar scientific code base which gets 10-15x wall-time improvement just by enabling compiler optimizations.
Re:Linux with “memory folios”: a 7% performance boost when c... by IntelMiner on Tuesday June 15, 2021 @07:05AM (#27513569)
Not a scientific codebase, but ffmpeg seems to benefit massively from optimization

I use it to encode my DVD and Blu Ray TV/movies to HEVC. Doing so reduces the filesize on DVD's by roughly 80% and Blu Rays by 60%

The downside is the encoding is an incredibly CPU intensive process. Hardware encoders like Nvidia's Nvenc or Intel's QuicSync look absolutely terrible and are a non-starter for archival storage

On a stock Fedora XFCE install, I would get roughly 0.5 FPS for a 1080p Blu Ray file (29.97 FPS at 1920x1080)

A Gentoo Linux installation with a global -O3 -march=native as well as LTO, PGO and Graphite enabled globally boosts it from 0.5 to roughly 1.3 FPS. Still slower than realtime, but an absolutely massive improvement

Re:Linux with “memory folios”: a 7% performance boost when c... by astrange on Tuesday June 15, 2021 @01:39PM (#27518764)
I suspect -march=native is the only thing doing any work here. The other optimizations are as likely to find compiler bugs as they are to improve things, once you get off well-tested paths.
Re:Linux with “memory folios”: a 7% performance boost when c... by bayindirh on Tuesday June 15, 2021 @02:22PM (#27519433)
I think it's the opposite. A good annotated code which can use SIMD instructions accelerates tremendously with -O3. -march=native -mtune=native generally improves things if you can saturate the cores with instructions, which can be measured by running perf record and looking to IPC & instruction retirement numbers.

When I was using Eigen on my code, the biggest performance boost came from -O3. -march and -mtune did minimal improvements on the systems which I've ran benchmarks on.

Re:Linux with “memory folios”: a 7% performance boost when c... by astrange on Tuesday June 15, 2021 @02:57PM (#27519903)
ffmpeg and associated projects do not rely on autovectorization, which pretty much only works on naive scientific code. They write their SIMD in assembly and don't need to care about compiler settings.
Re:Linux with “memory folios”: a 7% performance boost when c... by bayindirh on Tuesday June 15, 2021 @03:33PM (#27520381)
If you're compiling hand tuned assembly, -march and -mtune probably will have more effect when compared to compiling and optimizing C/C++ code.

OTOH, I'd like to underline that heavily optimized scientific code and libraries are neither naive (in terms of algorithmic complexity/implementation) nor straightforward :D

E.g.: This is how Eigen configures its internal vectorization parameters: https://gitlab.com/libeigen/eigen/-/blob/master/Eigen/src/Co...

Re:Linux with “memory folios”: a 7% performance boost when c... by ta988 on Tuesday June 15, 2021 @09:43AM (#27515423)
That's an amazing compression ratio, how is the quality?
Re:Linux with “memory folios”: a 7% performance boost when c... by IntelMiner on Tuesday June 15, 2021 @10:50AM (#27516341)
ffmpeg recommends a crf of 23 to preserve 1:1 for H.264 -> h.265

I do a probably overkill CRF of 20 just to be safe. But everything looks absolutely perfect, even blown up on my 65 inch (1080p) TV

Re:Linux with “memory folios”: a 7% performance boost when c... by bayindirh on Tuesday June 15, 2021 @11:12AM (#27516663)
Can you share your complete parameters, so I can do the same thing for my DVD archives & other stuff?

No hard feelings if you don't want to though :)

Re:Linux with “memory folios”: a 7% performance boost when c... by IntelMiner on Tuesday June 15, 2021 @10:46PM (#27524319)
Sure! This is the script I wrote for it. I use MP4 containers so I can import them into MAGIX (formally Sony) Vegas but it will iterate over an entire folder of MakeMKV files and convert them https://dpaste.com/DBPA9C59M
Re:Linux with “memory folios”: a 7% performance boost when c... by bayindirh on Wednesday June 16, 2021 @02:09AM (#27525401)
Thank you. It's greatly appreciated. :)
Re:Linux with “memory folios”: a 7% performance boost when c... by jorvi on Tuesday June 15, 2021 @12:08AM (#27511172)
I don't know if you would consider it tuning, but a low latency kernel is a massive UX improvement on consumer devices, where you don't want the UI to stutter and lock up under heavy load, and don't mind paying (could be wrong, top of my head) like a 5% throughout penalty.
Re:Linux with “memory folios”: a 7% performance boost when c... by rwmj on Tuesday June 15, 2021 @06:02AM (#27513154)
We just improved the performance of one copying tool by a factor of 2.9, by tuning the size of a memory buffer. (BTW a shout out to both Flamegraphs and hyperfine - both excellent tools for profiling and benchmarking respectively)
Re:Linux with “memory folios”: a 7% performance boost when c... by nitrogen on Tuesday June 15, 2021 @02:17PM (#27519371)
Tuning kernel/process priorities has made a big difference in my audio code. Tweaking the GPU and USB IRQ handler priorities let me do realtime audio together with realtime OpenGL data visualization and screen recording, where before the audio would skip or the visualization would hang.
Re:Linux with “memory folios”: a 7% performance boost when c... by MaxBarraclough on Tuesday June 15, 2021 @06:55AM (#27513464)
As someone with a limited knowledge of kernel stuff, does this mean Linux is likely to significantly outperform other comparable kernels like FreeBSD and OpenSolaris? Or are those other kernels keeping pace?
Re:Linux with “memory folios”: a 7% performance boost when c... by marcodiego on Tuesday June 15, 2021 @07:04AM (#27513564)
Considering the absolute dominance on HPC for a few years now, I don't think there is competition against Linux at the moment in terms of performance. Last time a non Linux system appeared in the top500 list was in June 2017.
Re:Linux with “memory folios”: a 7% performance boost when c... by hawk_ on Tuesday June 15, 2021 @09:24AM (#27515083)
I suspect that's more due to "popularity" or richness of the ecosystem than performance necessarily.
Re:Linux with “memory folios”: a 7% performance boost when c... by damagednoob on Tuesday June 15, 2021 @08:23AM (#27514224)
Isn't cost also a factor at play here?