News for hackers. Stuff that matters.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by a-dub on Sunday July 21, 2019 @02:09PM (#20492728)
If fsync() fails isn't valhalla lost anyhow? If you can't write things down because your pencil is broken, probably time to stop what you're doing and get a new pencil.

If the kernel can't flush dirty write buffers, maybe it's time to send up a flag and panic in the kernel itself?

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by jcranmer on Sunday July 21, 2019 @02:14PM (#20492764)
> If the kernel can't flush dirty write buffers, maybe it's time to send up a flag and panic in the kernel itself?

Being unable to write to a disk is a recoverable scenario, especially in some conditions. The most common cause of disk write failures is "someone yanked the flash drive out of the computer," and the recovery is "pop up a dialog telling the user to put it back in."

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by carbocation on Sunday July 21, 2019 @02:31PM (#20492872)
Another situation: a GCSFuse volume dealing with temporary networking errors.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by masklinn on Monday July 22, 2019 @02:08AM (#20495674)
Or NFS being NFS.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by tinus_hn on Monday July 22, 2019 @05:19AM (#20496412)
That would be a great improvement over the ‘tell the user he shouldn’t have done that’ we have now!
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by londons_explore on Sunday July 21, 2019 @06:51PM (#20494177)
I think there is a case for never throwing away dirty pages.

Ie. either the write eventually succeeds, or all other writes will block or fail until it does.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by tlb on Sunday July 21, 2019 @02:12PM (#20492751)
That can happen when you remove a removable storage device, like a USB drive. You don't want to panic the kernel for that.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by wruza on Sunday July 21, 2019 @04:31PM (#20493481)
FreeBSD lived with that up to circa 2010, afair. Few fast-pulling desktop linux trolls complained at the time, but nothing serious.

Anyway, if given such choice, as a user/admin I would prefer an instant fail with a panic message, rather than a corrupted 10gb online client database with no clear recovery path. I’m always scared by this kind of story when something fails to fail underneath and I even didn’t ever hear of it.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by Dylan16807 on Sunday July 21, 2019 @10:52PM (#20495058)
Isn't your data already corrupted most of the time when writes start to fail? If the drive is gone, a panic won't help or hurt, it will just ruin everything else you have open.

Redoing failed writes should be the method that ends in the least amount of corruption.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by wruza on Monday July 22, 2019 @04:16AM (#20496157)
Not going into details, at the end of the day one should have a plan to recover their value from a potential failure. By the means of version control systems, transaction logs, backups, mirrors, etc. But if a failure makes its way to a recovery storage undetected, the entire plan is doomed.

I don’t care if I lose few last hours of my local work, since this time is buffered in a project or is not precious at all (like game saves). That throws back me a little, but not anyone else. I also don’t really care about losing downloads or experimental files. But when it comes to a business database or a project repository, that value can be orders of magnitude bigger and can cost responsible one their job at the very least.

“Redoing failed writes should be the method that ends in the least amount of corruption.”

But in fact it led to a database corruption that was only fixed by some black magic. As far as I understood, they did manual rollback of failed writes. I wouldn’t blame anyone for subj bug even if met it by myself — writing low-level software is hard, the behavior was unintended, and both linux, xfs & pg are free as in beer and free of any charges. But if it happened and was not detected for longer than a recovery window allows it, costs were imminent. That’s why this trade-off is easy on a distance.

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by Dylan16807 on Monday July 22, 2019 @02:28PM (#20500789)
> But in fact it led to a database corruption

You're saying you had a system that did redo failed writes to disk, unlike other systems discussed so far, and it managed to corrupt your database anyway?

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by Gibbon1 on Sunday July 21, 2019 @03:38PM (#20493220)
Reminds me. On CPM, hit save when the drive door is open, so sorry. All gone.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by rodgerd on Sunday July 21, 2019 @03:55PM (#20493307)
So kernel panic on disk full errors? Because that's one of the scenarios causing data loss here - fsync throwing away data on a full disk and lying about having committed it.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by masklinn on Monday July 22, 2019 @02:13AM (#20495691)
In that thread, Craig Ringer and Thomas Munro notes most systems & FS won't report ENOSPC on fsync(), they'll do that on write():

> ENOSPC doesn't seem to be a concern during normal operation of major file systems (ext3, ext4, btrfs, xfs) because they reserve space before returning from write(). But if a buffered write does manage to fail due to ENOSPC we'll definitely see the same problems. This makes ENOSPC on NFS a potentially data corrupting condition since NFS doesn't preallocate space before returning from write().

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by limitthis on Sunday July 21, 2019 @07:09PM (#20494269)
no, the fs doesn’t overcommit storage like the memory system does.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by rodgerd on Monday July 22, 2019 @06:19AM (#20496677)
I don't think you understand the problem.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by a-dub on Sunday July 21, 2019 @02:52PM (#20492989)
mmm, maybe this is a little too aggressive there could be perfectly functional storage on the host.

maybe something like "make the mount read only and ensure all open fds on the device return nothing but error and then dump the dirty blocks to swap in some kind of recovery log that could potentially be applied to the disk"

Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by Upvoter33 on Sunday July 21, 2019 @08:29PM (#20494593)
no, some errors are transient. a retry is a reasonable thing to be able to do in such a case. this happens with storage on SANs for example, due to a network blip.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by a-dub on Sunday July 21, 2019 @02:27PM (#20492842)
if you've got removable disks on your servers, valhalla was lost long ago!
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by xyzzyz on Sunday July 21, 2019 @02:31PM (#20492870)
Huh? Hot swapping disks is extremely common.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by dboreham on Sunday July 21, 2019 @02:38PM (#20492913)
Depends on your definition of 'hot' (and also on your definition of 'disk'). If you mean 'FS volume under active write load' then no it isn't extremely common.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by a-dub on Sunday July 21, 2019 @02:38PM (#20492910)
the raid device remains stable and fsyncs don't fail in that case.
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by stonogo on Sunday July 21, 2019 @04:59PM (#20493630)
You don't use networked filesystems?
Re:Fsyncgate: Errors on fsync are unrecoverable (2018) by a-dub on Sunday July 21, 2019 @07:23PM (#20494322)
not on production db servers, no.