For this reason, you want an OOM killer to kill stuff way earlier than when you actually run out of RAM, because if you ever reach that point it is too late. I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects.
Along those lines, take a look at this: [1]
> Pressure stall information were added to Linux 4.20 as a way to quantify resource pressure in the system in a better way than the traditional load average. PSI aggregates and reports the overall wallclock time in which the tasks in a system (or cgroup) wait for cpu, io or memory.
> This release [Linux 5.2] lets users to configure sensitive thresholds and use poll() and friends to be notified when a certain pressure thresold is breached within a user-defined time window. With this mechanism, Android can monitor for, and ward off, mounting memory shortages before they cause problems for the user. For example, using memory stall, monitors in userspace like the low memory killer daemon (lmkd) can detect mounting pressure and kill less important processes before device becomes visibly sluggish. In memory stress testing psi memory monitors produce roughly 10x less false positives compared to vmpressure.
I haven't tried it yet, but it sounds promising.
[1] https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_I...
> If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when the do become busy, you won't have to wait.
> At least, that's the theory as I understand it. Whether or not it works in practice, IDK.
These are the ideas that I hoped the original post was going to be about.
Disk cache is a good one. I'm not knowledgeable enough about this, will the Linux kernel automatically start using RAM as disk cache if there's enough free space? Is it smart enough to prioritize disk cache over infrequently used pages and then know to discard the disk cache if it does need to page stuff back in from swap?
> will the Linux kernel automatically start using RAM as disk cache if there's enough free space?
Yes, exactly.
> Is it smart enough to prioritize disk cache over infrequently used pages and then know to discard the disk cache if it does need to page stuff back in from swap?
It is, and this is what's controlled by the vm.vfs_cache_pressure sysctl that the post discusses how best to configure.
The memory freed from a page that got pushed to swap might be better used for serving disk cache, for example.
If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when they do become busy, you won't have to wait.
At least, that's the theory as I understand it. Whether or not it works in practice, IDK.
> And using swap as some kind of "RAM emergency cover" does not make any sense to me. Personally, I have never had a case where I could gracefully recover a host that had started swapping.
I have successfully recovered hosts that began swapping, and even hosts that filled their swap. But it's beyond painful, and in most of the cases where I have a host doing such swapping, an OOM kill would have been a welcome reprieve.
> Second, I find it's much harder to detect degraded performance than it is to detect a dead process.
Agreed, and restarting something that's been OOM'd is simple enough too. Our hosts tend to stop reporting metrics when they start swapping, simply b/c so little is actually getting done, so they get labelled as "down". Linux also exposes a metric called "major page faults" (IIRC, it's "pgmajfault" in /proc/vmstat) that records the number of major (required service from disk) page faults; if the rate of that value is too steep for too long, that ≅ swap thrashing.