This happens to me quite often on Linux and I've always wondered if there's a way to fix it. Is there?

Usually this happens on Linux because an application is using too much memory and the kernel is spending tons of resources reclaiming memory to prevent a system OOM.

The system can easily slow to a crawl as memory accesses and allocations are constantly triggering reclaim and causing the system to thrash. The kernel has no way to know that it should prioritize the terminal you are trying to open to kill things rather than try its hardest to keep the offending application running.

The best way to prevent this, if you know an application will use lots of memory, is to run it in a memory cgroup. With a memory cgroup you can limit the applications total allowed memory, reserving some for the rest of the system.

Unfortunately, I don't know of a super nice CLI, but here's an example using cgcreate/cgexec (https://packages.debian.org/stretch/cgroup-tools):

    $ # Create a memory cgroup named 'mygroup', allowing $USER to change the parameters and run tasks in the cgroup.
    $ sudo cgcreate -a $USER -t $USER -g memory:mygroup
    $ # Set the memory limit to 10MB
    $ echo 10000000 > /sys/fs/cgroup/memory/mygroup/memory.limit_in_bytes
    $ # Run a program in the cgroup
    $ cgexec -g memory:mygroup ls

> The kernel has no way to know that it should prioritize the terminal you are trying to open to kill things rather than try its hardest to keep the offending application running.

This is one of the problems multitask OS's were invented to solve. The answer is always let other tasks run too.

man ulimit?

ssh to the box then kill the process.

I've often been unable to connect to SSH on a host for lack of available CPU/memory to spawn the SSH receiver process + the shell.

It's especially a problem with AWS's tN.micro instances: when something thrashes them for long enough, they ratchet down to such a low CPU allocation that it will likely take hours before SSHD gets context-switched to enough times to start your process. By then, all sorts of timeouts will likely have killed your session.

That's worth making persistent noise about, I think. Because you just described something that can be accurately described as a type of DoS.

Considering the number of AWS credentials out there, I can see malicious users logging in, forkbombing the instance, getting it to throttle back, and then walking away whistling until the person with dashboard login wakes up and can reboot the machine.