Saves a lot of hassle with manually attaching GDB.
* catching of NULL pointer references (ie. transforming SEGV into catchable userspace exception) * testing whether some in memory data have been changed/accessed (ie. mprotect(), wait for SEGV, set flag, un-mprotect(), return) * paging in userspace (eg. across network)
All that seems obscure (and is mainly useful for virtual machines and such), but it is not so rare to find process that actually does this (usually because of some library, or because it is some kind of virtual machine)
I think it was Oracle.
If I remember correctly we had to get the vendor involved
This happened after an unclean shutdown (power failure). The disk were RAID, but the cache battery was dead so there was some corrupt data written to the disk on block level.
THis could happen too on a kill -9.
At the very least, you need to be prepared for a possibly very long replay of logs. Also, a huge number of people run databases in a configuration that doesn't make those guarantees. For example, many people will run mysql (esp. less performant slaves) with innodb_flush_log_at_trx_commit = 0 for performance with the understanding that a failure might require manual fixes.
Losing a transaction that was committed upstream can lead to a bunch of manual fixes in replicated database clusters. What happens if rows inserted in that lost transaction are eventually updated? Replication breaks, you get paged, and now you're faced with either manually fixing the DB consistency error or by rebuilding the whole slave. Woe be unto you if this host is a replication hub with a bunch of slaves hanging off it.
Edit: Especially when you have multiple systems that should be consistent. Deleting a user from the DB, but being killed before it can be removed from our auth system, for instance.
Postgres has special code to account for hard kills: it has a SYSV shared memory segment, to which every child attaches. If the parent dies, the children don't have a good way to know, so they might keep running. If you try to start a new postmaster and it sees that there are still processes attached to the shared memory segment (shm_nattch), it will fail to start.
Were it not for that code, it would allow you to start a new parent process, leading to chaos as two sets of backends were accessing the same files and shared memory without knowing about eachother.
If you're a developer, before you kill -9 a program send SIGTERM (ie kill without args or kill -15). If the program does not respond, run gdb -p <pid> and then "thread apply all bt" before killing it. At the very least, you should get a good idea of why it was not responding to other signals.