News for hackers. Stuff that matters.
Re:File consistency (2015) by pletnes on Friday March 30, 2018 @04:27AM (#16714610)
Looks to me like yet another reason to use sqlite instead of «flat files». E.g postgres seems like a many orders of magnitude increase in complexity.

Possibly related: We’ve been using hdf5 for a lot of data storage at work (raw image data). I often discover corrupt files, even though we (think we) are flushing files etc. I’d love to see some work on reliability there, but it’s hard to know if the article is relevant to those issues.

Also, what happens when you’re on RAID? Even more assumptions out the window I’d imagine?

Re:File consistency (2015) by zejn on Friday March 30, 2018 @06:16AM (#16714979)
sqlite also uses fsync, which at least on Linux has corner cases where errors don't even get reported. When fsync does report error, you should panic, as even if the subsequent fsync returns ok, that does not mean the data for which there was an error was actually written, in fact the opposite is true - kernel clears the dirty bit on non-written data thus neither the application nor kernel know which data is not written.
Re:File consistency (2015) by nordsieck on Friday March 30, 2018 @12:41PM (#16717618)
> sqlite also uses fsync

At least sqlite has done the hard work of figuring out how to be as reliable as possible.

At some level, writing to disk is a bit like using crypto. It's a lot easier to get right if you use well tested libraries that only offer high level primitives.