News for hackers. Stuff that matters.
Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by znpy on Saturday March 13, 2021 @07:11PM (#26451454)
I have some questions.

I started reading about systemd-oomd, then followed the link to its manpage (at https://www.man7.org/linux/man-pages/man8/systemd-oomd.8.htm...) then followed a link from there to this page: "In defence of swap: common misconceptions" at https://chrisdown.name/2018/01/02/in-defence-of-swap.html).

The first line of the tl;dr is: "Having swap is a reasonably important part of a well functioning system. Without it, sane memory management becomes harder to achieve."

Now, afaik linux as a kernel is pretty much designed to have swap memory and really not meant to be ran without it. Except disabling swap is required to run kubernetes, so much so that the kubelet will plain refuse to start if it detects swap memory enabled.

Why?

Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by aidenn0 on Saturday March 13, 2021 @11:13PM (#26452750)
Linux works fine without swap. It also works fine with swap. It does not work fine under memory contention regardless of swap, but it is much, much worse if you have many GB of swap on a spinning disk.

Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process.

It used to be that the recommendation for swap was 2x ram. This recommendation was from the days were single-digit mega bytes of ram was common, and also from when linux core-dumps were both expected, and stored to swap.

Fast forward to 2010ish. You might have a 64GB or 128GB SSD, but you probably also have a spinning drive. you have 8 or 16GB of ram, so your 16-32GB swap would be 1/4 the SSD, plus you heard that putting swap on an SSD wears it out faster, so you put a 16-32GB swap file on the spinning drive.

Lets say you are developing software and you accidentally allocate in a tight loop. At some point the majority of the pages in ram not from the malloc loop end up in swap, so you try to do something to kill it, but Xorg and xterm both have their pages scattered across spinning drives, and your program is allocating memory faster than your machine saps in. If you are patient enough to not hard reset, an hour later, swap fills up and the OOM killer starts killing programs, and probably (but not definitely) kills the buggy malloc loop before it kills something more vital.

You are annoyed, so disable swap and do the same thing: the OOM killer kills probably just your web browser (which is virtually indistinguishable to a malloc loop with Web 2.0), and the malloc loop. The system recovers in minutes. You now swear off ever running with swap again.

Today: If you have a spinning disk 1GB or swap on it is fine, but zram is another good alternative. If you have slow and/or fragile SSD storage (e.g. sdcard or eMMC), zram is a good option. If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.

Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by kalleboo on Saturday March 13, 2021 @11:45PM (#26452866)
> If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself

Not sure about Linux, but under macOS I had a bug that was leaking uncompressed 12 MP photos in a loop. I was browsing the web while waiting for my process to run and I got up to something like 150 GB of swap used (macOS uses dynamically growing swap files) before I even noticed that there was something going on.

Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by VLM on Sunday March 14, 2021 @10:45AM (#26455836)
> It used to be that the recommendation for swap was 2x ram.

It always boiled down to never allocate more swap than you're willing to wait, and memory capacity multiplied over the years far faster than real world drive thruput.

So I was willing to slow everything down to use 8 megs of swap in 1994 in order to have enough virtual memory to compile a kernel, and in 2021 I'm still willing to slow everything down to use 8 megs of swap, in fact drive thruput has maybe quintupled since then, maybe, so I'd be willing to slow my system down into using 32 megs of swap in 2021.

The problem is 8 megs of swap in 1994 dramatically increased my horizons and I could use it to compile the kernel and stuff like that. In 2021, 32 megs of swap just makes my web browser OOM kill about 100 milliseconds later so why bother?

Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by stouset on Sunday March 14, 2021 @12:46PM (#26456708)
As I understand it, swap is used even when the system isn’t under direct memory pressure.

If the kernel sees that some memory is barely, if ever used, it will swap it out preemptively. This gives you more memory available for both programs and temporary caches.

As such it’s still advantageous to add much more swap than this, since it effectively gives you extra free space in RAM at essentially no cost.

Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by awild on Sunday March 14, 2021 @08:41AM (#26454846)
> I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.

We are running a distributed in-memory olap database. All nodes have 500gb ram used basically at capacity. One node has 1.5T of nvme swap (initially just for testing but it works well) so it hosts a shard and the manager/reduce node. We've tested forcing the nodes onto swap and saw a x10 performance decrease, but our implementation is pretty shit when it comes to memory access patterns so it's not too bad. The collocated node is slightly slower in terms of performance but doesn't impact total response by a meaningful margin.

Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by hinoki on Saturday March 13, 2021 @11:57PM (#26452911)
> Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process.

Why can you not get swap usage? It’s in smaps, right?

Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by aidenn0 on Sunday March 14, 2021 @02:42PM (#26457655)
Oh smaps is newer than my knowledge. I did since more research and your still can't limit over RSS+swap with ulimit though. There's a chance you could with cgroups though as there's lots of features there.
Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by geofft on Saturday March 13, 2021 @07:17PM (#26451502)
This is one of the primary reasons the Kubernetes folks are working on letting you run with swap! See recent discussion in https://github.com/kubernetes/kubernetes/issues/53533 .

The historic reason Kubernetes didn't let you run with swap is that it's complicated from a resource management perspective - it's not clear whether each pod should be requesting some amount of swap the same way it requests some amount of RAM, and also the container runtimes didn't really have support for that (largely because you need cgroupv2 to have a hope of that sort of per-cgroup swap resource control working at all). But it's not an insurmountable problem, and it sounds from that ticket like they recognize that letting systems have swap enabled for the sake of a userspace oomd is an important use case.

Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by xyzzy_plugh on Saturday March 13, 2021 @11:01PM (#26452695)
Why is Linux not meant to be ran without swap? In 2021? I've never heard of this. I've operated massive clusters of thousands of hosts with no swap enabled with zero hiccups.

Swap comes with a lot of caveats -- when a performance-critical process begins swapping it can on occasion be more useful to just let it OOM instead.

My personal desktop has 64GB of memory, with no swap enabled. Why would I want swap it I don't need more than that?

Re:Systemd 248 RC3: systemd-oomd is now considered fully sup... by axaxs on Saturday March 13, 2021 @11:46PM (#26452872)
You don't need it. Linux works fine without swap, given ample memory.

In my experience, even with ample swap, when Linux gets memory starved it goes haywire anyways.

I'm an ex linux sysadmin, and haven't run swap on my personal systems in years.