What you are probably looking for is the magic SysRq key [1] If you enable sysrq then you can send key-sequences directly to the kernel to do things like emergency file system sync, emergency dismount of filesystems, reboot, panic and so on. Not all keyboards have a SysRq key so you may have to map something as that key.

    grep sysr /etc/sysctl.conf
    kernel.sysrq = 1

One can also re-map cntrl+alt+del to do anything if the system is not entirely hung and that configuration incantation varies by what init system your machine is using.

Another thing to research is tuning options like vm.min_free_kbytes, vm.swappiness, vm.lowmem_reserve_ratio, vm.user_reserve_kbytes, vm.admin_reserve_kbytes, vm.overcommit_ratio and vm.overcommit_memory in /etc/sysctl.conf to swap early, and reserve more memory for the kernel based on how much installed RAM you have available. Just setting vm.overcommit_ratio to 0 can solve some hung systems for a number of people at the expense of some applications not launching when memory is low. Another thing to research if you do not have enough RAM is zswap, especially useful to reduce writes to SSD's. To see if your kernel has zswap support:

    modprobe configs && zcat /proc/config.gz | grep -i zswap|grep -v ^#

[1] - https://www.kernel.org/doc/html/v4.15/admin-guide/sysrq.html

Ah, thank you!

I use SysRq to REISUB my machine — but I haven't found the SysRq key on Macbook keyboards; and haven't had the nerve to enable SysRq on friend's computers (which I often end up troubleshooting), since I didn't know much about it, what the security implications are, and whether 1 is an advisable value for kernel.sysrq

Those sysctl options could make a big difference; I've wondered about something like that, but assumed the optimal default was built in. Thank you! I'll try tinkering with those values.

There was a time I found some organizations to be paranoid about enabling it because it was mentioned in either CIS or NIST hardening but it ignores the fact that the same information can be gathered with other tools so I suppose it depends on if you have to follow a standard that still says sysrq is a risk and if your SOC1/SOC2 documentation refers to those standards as part of your audits. But in reality if that were a risk then there are a myriad of other things that would have to be disabled as well and some of those things require recompiling the kernel to disable.

If your system has frozen and you can't get a terminal with ctrl+alt+F1 (or F2 etc) then consider it dead, and hit the hardware reset or power.

You do have a hardware reset button?

top or htop in the terminal

Do you know of a more low-level solution for when when my system is frozen, and the keystroke to open terminal (and run htop or "killall APPNAME") doesn't do anything?

Try ctrl+alt+f1 to switch to the first terminal (leave desktop)

ctrl+alt+f7 to go back to desktop

this will work only if it's Xorg that has crashed.

For a kernel panic or lockup I think it's a combination roughly similar to:

1. Ctr-Alt-PrintScreen

2. continue to hold Alt

3. R-I-S-U-B as a sequence, to make certain things happen and like syncing the file system

After you get to tty using ctrl alt F. killall sends a TERM signal by default. You could send a KILL signal using "killall -KILL APPNAME". You could also run top using the nice command: "nice -20 top" in order to get the highest priority possible for the top process.