djb's old daemontools suite had an interesting solution to the "how to log failure when you can't write to a log" problem: a persistent child process with some space reserved in its argv, where it can stash some error messages which you can see with "ps" even when disks have failed.

http://cr.yp.to/daemontools/readproctitle.html