Being unable to write to a disk is a recoverable scenario, especially in some conditions. The most common cause of disk write failures is "someone yanked the flash drive out of the computer," and the recovery is "pop up a dialog telling the user to put it back in."
Ie. either the write eventually succeeds, or all other writes will block or fail until it does.
Anyway, if given such choice, as a user/admin I would prefer an instant fail with a panic message, rather than a corrupted 10gb online client database with no clear recovery path. I’m always scared by this kind of story when something fails to fail underneath and I even didn’t ever hear of it.
Redoing failed writes should be the method that ends in the least amount of corruption.
I don’t care if I lose few last hours of my local work, since this time is buffered in a project or is not precious at all (like game saves). That throws back me a little, but not anyone else. I also don’t really care about losing downloads or experimental files. But when it comes to a business database or a project repository, that value can be orders of magnitude bigger and can cost responsible one their job at the very least.
“Redoing failed writes should be the method that ends in the least amount of corruption.”
But in fact it led to a database corruption that was only fixed by some black magic. As far as I understood, they did manual rollback of failed writes. I wouldn’t blame anyone for subj bug even if met it by myself — writing low-level software is hard, the behavior was unintended, and both linux, xfs & pg are free as in beer and free of any charges. But if it happened and was not detected for longer than a recovery window allows it, costs were imminent. That’s why this trade-off is easy on a distance.
You're saying you had a system that did redo failed writes to disk, unlike other systems discussed so far, and it managed to corrupt your database anyway?
> ENOSPC doesn't seem to be a concern during normal operation of major file systems (ext3, ext4, btrfs, xfs) because they reserve space before returning from write(). But if a buffered write does manage to fail due to ENOSPC we'll definitely see the same problems. This makes ENOSPC on NFS a potentially data corrupting condition since NFS doesn't preallocate space before returning from write().
maybe something like "make the mount read only and ensure all open fds on the device return nothing but error and then dump the dirty blocks to swap in some kind of recovery log that could potentially be applied to the disk"
If the kernel can't flush dirty write buffers, maybe it's time to send up a flag and panic in the kernel itself?