Under what scenario will a server start swapping due to load, only to continue performing as it should? It's hard for me to come up with a real world scenario where performance doesn't drop from peak, leaving a backlog of requests rapidly growing, which then compounds the problem.

Either way you handle it, the server is toast. So the real solution is to start dropping requests or downgrading service in some way to make sure the server never reaches OOM.

It definitely is important that the person signing the cheques knows how much capacity they're paying for, but that goes two ways. If you underprovision, they must understand that if they ever get mentioned in the NYT, their site will likely go down. It's up to them to balance the risks.

Take something as simple as SSHD, NTPD, CROND or any other of the many background daemons that get used once or twice a day. They only need a few bytes of themselves in active memory to sit waiting on a socket connection or on a timer. The rest can be paged out for the actual application you're using the server for. It doesn't even matter if these services take a few milliseconds to get paged back in when needed since things like SSHD are working over the network (many orders of magnitude slower than hard drive access).

There's basically no reason not to have at least a small amount of swap. There's never been a benchmark showing no-swap as faster.

It should also be mentioned that things like mmap()+MAP_PRIVATE on a file will flat out fail if the file is larger than free_memory+swap. It's a common, easy and fast way to work with large files where the portion of the file you're working on gets paged in and out as required. Turn off swap and you break this functionality. You're basically limiting what you can do on your system with no performance gains.