BTDT, a lot of times. In the meantime I notice the system is slowing down, but I can't kill the guilty process because the system is too slow to ssh in. The ssh client times out while the ssh server is trying to obtain the pages it needs.

Swap is what turns the deficient OOM killer from a minor to a major problem.