The memory freed from a page that got pushed to swap might be better used for serving disk cache, for example.
If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when they do become busy, you won't have to wait.
At least, that's the theory as I understand it. Whether or not it works in practice, IDK.
> And using swap as some kind of "RAM emergency cover" does not make any sense to me. Personally, I have never had a case where I could gracefully recover a host that had started swapping.
I have successfully recovered hosts that began swapping, and even hosts that filled their swap. But it's beyond painful, and in most of the cases where I have a host doing such swapping, an OOM kill would have been a welcome reprieve.
> Second, I find it's much harder to detect degraded performance than it is to detect a dead process.
Agreed, and restarting something that's been OOM'd is simple enough too. Our hosts tend to stop reporting metrics when they start swapping, simply b/c so little is actually getting done, so they get labelled as "down". Linux also exposes a metric called "major page faults" (IIRC, it's "pgmajfault" in /proc/vmstat) that records the number of major (required service from disk) page faults; if the rate of that value is too steep for too long, that ≅ swap thrashing.
For this reason, you want an OOM killer to kill stuff way earlier than when you actually run out of RAM, because if you ever reach that point it is too late. I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects.
Along those lines, take a look at this: [1]
> Pressure stall information were added to Linux 4.20 as a way to quantify resource pressure in the system in a better way than the traditional load average. PSI aggregates and reports the overall wallclock time in which the tasks in a system (or cgroup) wait for cpu, io or memory.
> This release [Linux 5.2] lets users to configure sensitive thresholds and use poll() and friends to be notified when a certain pressure thresold is breached within a user-defined time window. With this mechanism, Android can monitor for, and ward off, mounting memory shortages before they cause problems for the user. For example, using memory stall, monitors in userspace like the low memory killer daemon (lmkd) can detect mounting pressure and kill less important processes before device becomes visibly sluggish. In memory stress testing psi memory monitors produce roughly 10x less false positives compared to vmpressure.
I haven't tried it yet, but it sounds promising.
[1] https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_I...
> If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when the do become busy, you won't have to wait.
> At least, that's the theory as I understand it. Whether or not it works in practice, IDK.
These are the ideas that I hoped the original post was going to be about.
Disk cache is a good one. I'm not knowledgeable enough about this, will the Linux kernel automatically start using RAM as disk cache if there's enough free space? Is it smart enough to prioritize disk cache over infrequently used pages and then know to discard the disk cache if it does need to page stuff back in from swap?
> will the Linux kernel automatically start using RAM as disk cache if there's enough free space?
Yes, exactly.
> Is it smart enough to prioritize disk cache over infrequently used pages and then know to discard the disk cache if it does need to page stuff back in from swap?
It is, and this is what's controlled by the vm.vfs_cache_pressure sysctl that the post discusses how best to configure.
Putting rarely used (or never to be used again) pages in swap frees up physical RAM to be used as disk cache, which can improve performance for file access. If you have enough RAM to cover all application memory, and cache your entire filesystem, there is no gain to be had.
I agree that slowing down low memory scenarios is silly though. In many cases killing a process is easier to deal with than unexpected poor performance. The slow death of out of memory systems doesn't seem to be unique to having a swapfile though; Linux seems to discard any recoverable pages (such as application / library code backed by files on disc) in the run-up to OOM, resulting in thrashing.
Ideally I'd want to have swap, but automatically kill processes when performance deteriorates due to swapping, rather than when the system is strictly out of memory.
The only way that makes sense is if there is some cache the kernel has which is faster than RAM. BUT the kernel cannot move things between this cache and RAM, it can only treat the cache + RAM as a single blob, so to free up space in the cache the kernel has to move pages to disk. So by having some swap space the kernel can move hardly ever used pages to disk which is the only way to free up space in this cache.
It's possible that what I just described really exists, I don't know that much about the internals of how the kernel manages memory. But if that is what's happening then THAT is the detail that would have been useful in this post.
I had a small, very lightly loaded VPS that had no swap. Occasionally, it would not start all its services on boot, in particular the db, due to OOM.
Adding a small amount of swap allowed it to boot cleanly. The swap was not subsequently used.
The alternative was to upgrade the server RAM at double the monthly cost.
One thing I think the author was getting at, but never explicitly stated is that we spec most things for the average expected workload with a bit of headroom--not the peak workload since that is very difficult to assess. Peak workload might also be few and far between and not worth paying for upfront. Swap allows you to deal with peak workloads more gracefully for many applications.
Having your process randomly die isn't a big deal with stateless processes (which is a more and more common design), but really sucks for stateful things. Not just the possibility of lost/corrupt state, but process startup and loading of state can take a long time.
We have a few servers that don't have swap and I was hoping to find an excuse to add it in this article. Instead, it reenforced that decision because of type of things we run on it.
This is almost always false in practice though. Peak workloads means more than just lots of memory, it means high CPU and high IO (disk and network). If you under-spec your servers such that they need to hit swap during peak loads they are going to generally cause more problems than if they just die. A server dying is something you have to prepare for and deal with anyways, and is much simpler to plan for than the weird latency issues you'll get with a server hits swap.
Think of it as similar to Erlang's fail-fast model. Better for things to just die and be recovered than to continue on in a bad state.
If your hardware is underspeced for the workload, there's still a business case to be made before you buy new hardware. Sometimes the preferred failure mode is to slow down, other times it's preferred to die and restart. Hardware speced for peak loads can sometimes cost multiple times otherwise appropriate hardware...assuming you even know what peak loads will be.
I have one class of application that is fairly idempotent, we don't have hard drives on these machines, so we don't use swap and expect it to die if it exceeds the amount of memory requested. This article reenforced this.
I have another class of application that queues up things to process. The specs work on the current hardware 99% of the time, but on holiday weekends when large projects are delivering things slow down because work is submitted and people leave so nobody cares. Reloading the state from disk can take 20m+ and without swap if you were to exceed memory you're dead in the water until you purchase more hardware. Our contingency hardware has identical specs as the original.
But that still relies on uncorrelated failures.
Otherwise that seems like an orthogonal discussion to swap vs. OOM killer. It's something to be aware of and you need circuit breakers in place but I don't think it's an argument for letting hosts swap instead of having the OOM killer deal with it.
Using swap space may make sense for a traditional physical, stand-alone server, where having it fail quickly doesn't really help. Or maybe for a workstation.
And using swap as some kind of "RAM emergency cover" does not make any sense to me. Personally, I have never had a case where I could gracefully recover a host that had started swapping.
First, my production servers are not doing many other things besides being production servers. It's not like they are running a bunch of unnecessary services, and if I kill some then my application can recover. If a process is out of control then it's almost assuredly something important that I'm going to have to kill anyway.
Second, I find it's much harder to detect degraded performance than it is to detect a dead process. It's very, very easy to have a health check that will detect a host who has stopped listening and drop that host from the LB. And alerting on that scenario is very easy as well. The alternative is a host that's operating in a degraded state, which I need to detect with more sophisticated health check + alerting, and in the end my resolution is just going to be to kill everything anyway.
In a properly designed HA environment the loss of a host should be no big deal. Architecture should be focused on making sure a host goes down ASAP if it's having problems, not letting it survive in some kind of zombie state.