1. Create a 1 GB file of "nothing" (e.g. dd if=/dev/zero of=/path/meh.bin bs=1000 count=1000000) on each filesystem (exception: /boot, which is typically 256 or 512 MB).
2. Configure each filesystem to reserve 5% of space for the root user. This is typically the default, but I do it explicitly just to make sure.
When the first "low disk space" warning comes in, that 5% can be adjusted down to 1% to free up some space (on today's large filesystems, this 4% usually amounts to quite a bit of space). This buys you enough time (hopefully) to expand logical volumes, etc.
If you manage to about run out of space after freeing up that 4%, you have one more chance: delete that 1 GB file.
Another favorite of mine is: looking at the right file, but on the wrong host (for example when connected via ssh over a fast network, you simply forget that you're in a remote shell).
I've heard of people doing things like having a terminal background colour escape sequence in their login scripts as well, so it's immediately obvious if you're on a production host.
[1] http://manpages.ubuntu.com/manpages/natty/man8/molly-guard.8...
Now, shutdown and rm are the worst to add to the mix.
Hint: it was only once.
At my last job, we had a MySQL instance that kept falling over. It turns out that someone was dividing a table up with tons of partitions, and MySQL creates a new data file for each partition. The MySQL instance created several hundred file descriptors while trying to open each one of these files, and "mysteriously" crashed.