News for hackers. Stuff that matters.
Re:Durability: Linux File APIs by mehrdadn on Monday October 12, 2020 @05:33PM (#24759672)
> How can software possibly be expected to ensure what a physical device returns after the system restarts?

By giving instructions to the device that the device understands and provides guarantees for...

Re:Durability: Linux File APIs by formerly_proven on Monday October 12, 2020 @06:40PM (#24760285)
On hard drives, even SATA ones this is mostly sorted out since a few years (Linux doesn't use FUA over SATA, leaves write-back drive cache enabled, and just issues a full cache flush, which the big majority of drives respect, more or less). SSDs, less so, fsync is a rather relative affair on some of them.

IMHO, from having spent a fair amount of time on this, durability is a "fluid" concept, especially on desktop hardware (Microsoft had the right idea when they built Win Update on TxF), so if the data is important enough to worry about it, you should architect the system in a way that the durability of a fildes or drive is irrelevant to the overall system.

Re:Durability: Linux File APIs by mehrdadn on Monday October 12, 2020 @07:02PM (#24760509)
Right, but that's only because drives are lying at that point (or implemented so incompetently they have no idea they're lying...). Linux is trying to force liars to do what it wants anyway, and hence as a practical matter, programs need to worry about this too. This doesn't mean the spec is somehow broken or that honest (and competent) implementers couldn't guarantee durability if they wanted to.
Re:Durability: Linux File APIs by chrisseaton on Monday October 12, 2020 @05:35PM (#24759693)
How can the device provide those guarantees? If the system crashes hard in a power surge and the head crashes there's not going to be any use in claiming 'but the API guaranteed it'. There's no true durability.
Re:Durability: Linux File APIs by tzs on Monday October 12, 2020 @08:38PM (#24761153)
You make it a requirement for some certification that the device maker wants to get.

I once wrote the Novell NetWare drivers for a SCSI host adapter vendor, and part of Novell’s certification testing was to run I/O heavy tests on a system that would randomly cut power to the drive or the computer and make sure that this did not corrupt things beyond what NetWare could fully recover from.

Hardware vendors like host adapter makers and drive makers wanted Novell certification, so made sure that was possible.

They might also provide settings that sacrificed that for more performance, but they would make sure you could get decent performance using the full durability settings.

Re:Durability: Linux File APIs by mehrdadn on Monday October 12, 2020 @05:41PM (#24759743)
> How can the device provide those guarantees? If the system crashes hard in a power surge and the head crashes there's not going to be any use in claiming 'but the API guaranteed it'. There's no true durability.

By your logic there is no such thing as a guarantee in this world, period. A meteor could crash into Earth and wreck everything.

Re:Durability: Linux File APIs by chrisseaton on Monday October 12, 2020 @05:43PM (#24759763)
> Very astute observation.

You're being snarky. That's not allowed in the rules here and it's unkind even if it wasn't.

> By your logic there is no such thing as a guarantee in this world, period.

So what does the guarantee really mean?

That the data will be on disk... as long as the crash isn't too hard? What do you usefully do with this guarantee? How is it better than no guarantee?

Re:Durability: Linux File APIs by mehrdadn on Monday October 12, 2020 @05:53PM (#24759868)
Okay, I removed that phrase, but what do you want me to say here? Entire disciplines have devoted decades of research into providing useful guarantees and yet you're here observing that no human could possibly prevent some disasters... and using that logic to just trash everything and everyone's associated work along the way. I mean, you're clearly technically right that there is no such thing as an absolute guarantee, but... what am I supposed to do with this assertion? Find it insightful? Give up on the notion of a guarantee? Declare everything is useless? What response were you hoping for that you're still not getting?
Re:Durability: Linux File APIs by chrisseaton on Monday October 12, 2020 @06:04PM (#24759974)
Why do you think the phrasing 'shall request ... is intended to' means there? It's a trapdoor for 'actually guarantees nothing', right?
Re:Durability: Linux File APIs by mehrdadn on Monday October 12, 2020 @06:14PM (#24760060)
It means: "In case we didn't get the point across in the technical language above, this is what we're trying to say in plain English."

It does not mean "we're just writing this superfluous text to clock in extra hours or pass a word count threshold".

Re:Durability: Linux File APIs by chrisseaton on Monday October 12, 2020 @06:27PM (#24760190)
> It does not mean "we're just writing this superfluous text to clock in extra hours or pass a word count threshold".

Where have I said I thought it was about writing time or a word count? You've imagined that.

Re:Durability: Linux File APIs by mehrdadn on Monday October 12, 2020 @06:50PM (#24760399)
Well I can't guess what else you imagined it to mean besides those, but I hope it's clarified now.
Re:Durability: Linux File APIs by chrisseaton on Monday October 12, 2020 @06:51PM (#24760409)
Not really - the language still seems super vague - intentionally so and thus useless. But thanks for trying despite your for some reason bitter snark and sarcasm!
Re:Durability: Linux File APIs by mehrdadn on Monday October 12, 2020 @07:03PM (#24760529)
I'm still lost what kind of a response you were hoping for that you didn't already receive, but you're welcome.
Re:Durability: Linux File APIs by icedchai on Monday October 12, 2020 @11:47PM (#24762073)
I take it to mean that absent some unexpected, external event (like a head crash, flood, meteor, EMP, etc...) data will be on disk.
Re:Durability: Linux File APIs by Dylan16807 on Tuesday October 13, 2020 @01:40AM (#24762610)
The device would guarantee that the data is safe as long as it doesn't get physically damaged. That's an extremely useful guarantee. Power outages are common and almost never cause physical damage.

A flash device that isn't designed carefully enough could lose multiple megabytes of semi-random sectors if it gets interrupted at the wrong time, as a consequence of giant erase blocks.