Maybe if the kernel behaved optimally it wouldn't be any different, but empirically it seems like it does help.

My guess would be that if it's all RAM, then the kernel will happily use all of it for IO buffers, etc. and make no attempt to reduce its usage until actually necessary. If some of it is swap, then the kernel will (over time) try to reduce the size of its IO buffers until swap is no longer needed, but that de-allocation doesn't need to happen immediately, or block the rest of the system whilst it does happen.

Maybe there is a way to have that "high water mark" without needing swap - reserve 500M to be only used in "emergencies" - I don't know the details of the linux kernel well enough to know if that's a possibility.

Now I'm suddenly wondering if this isn't really easy after all (or rather: should be really easy if all was well). If Linux behaves better when it has some slack space past the normal RAM that it doesn't like to put stuff in but will when necessary (i.e. swap), let it do the exact same thing to the last portion of your RAM. In other words: make a swap file of, say, 10% of your memory size, and put it in a tmpfs. Linux probably doesn't want that I believe, but it would, if working, give the results you want.

> make a swap file of, say, 10% of your memory size, and put it in a tmpfs

You're looking for zram, I believe.

Of course, having more ram would be more useful than having swap in general, but swap provides for a small cushion in case your usage grows over time.

If your load fits into 14gb and you get 16gb of ram and add 0.5gb of swap, when you see swap start to fill, you know it's time to look for memory leaks and/or get more ram. If you have a bigger memory leak / burst allocation, you have a chance of being able to connect in and shut the thing down cleanly.

There are some issues with kernels swapping out 'the wrong' pages, and filling swap early. But, assuming that's tuned, I'm not aware of a better indicator of you need more ram than your swap is full / swap i/o is high.