So I think that fear is valid that they'll implement the API without the guarantees.
Myself I'm annoyed with the reckless push always to remove guarantees in return for 'performance'.
Filesystems have other horrors that you learn about during testing. I had one test case where it would take ext4 80 seconds to write out an 8MB file after a fresh mount of an 8TB filesystem. Free space was fragmented in just the right way that we hit the single threaded reading of block groups and bitmaps in the kernel and it took forever to get that data off the disk array.
What benefit does that property provide? Other than helping to deal with random memory corruption bit-flips, I'm having trouble understanding why state machines (in hardware, or any level of software) should ever be expected to handle arbitrary memory manipulation outside of the rules of the state machine.
Seems to me the bigger issue is the deification of POSIX and UNIX. There's a stupidly large contingent of people that think that they are flawless and must be followed unthinkingly.
It's not what I would actually call designed. The closest analogy I can think of for non-c programmers is that posix is like if we decided that people should only make websites using javascript that was mutually interpretable by IE6 and Netscape navigator and we occasionally made updates every decade or two.
There's really nothing particularly noble or good or correct or elegant about all the API calls with implementation defined semantics. At best it's a necessary evil for compatibility. You can admire the cleverness required to get a single simple c codebase that works correctly on multiple operating systems and future operating systems that conform to posix in creative ways, but only in the way I admire those old school zines that are simultaneously a PDF and a jpeg and a shell script. Clever and ingenious but not actually good engineering design.
Like signals, for instance. Okay idea when users are writing C from scratch to implement simple things. Horrible pain in modern times, with threads and libraries not playing well with signals.
Answer: It lies with the people wanting a fix!
Moral of the story: backwards compatibility is a menace.
Have a function that is documented to implement a highly specific contract, like "writes all pending blocks of this file to disk, and ensures they'll be there if there's a crash after the function returns".
I don't know how much this tracing has improved in various kernels since 2009, but I do know that I would want them to benefit fsync(), not to make a new function with a subtly different contract.
- Unit files. You create, you write, you close, and then others can read the new version. Until the original writer closes and gets a good close status, nobody else can read it. If the program aborts or the system crashes before closing cleanly, the file reverts. All readers see a fully written file. This is the default. It's what most programs need. Replacing a file by creating a new one on top of it is both permitted and an atomic operation. UCLA-LOCUS and some IBM systems that followed worked that way.
- Temporary files. Disappear on system crash.
- Log files. Append-only. All readers are guaranteed to see an end of file position that corresponds to the end of a previous write. Usually the most recent write, but buffering may make log file reading run a little behind.
- Managed files. Read, write, share. Write operations return two completions, probably via some async mechanism. The first completion means "buffer contents taken". The second completion means "committed to storage that will survive a crash". Database systems would use this, but few other programs would bother.
This tells the database what it really needs to know - when is the data safe? That tells the database when it can commit a transaction. The database can do other things, including more I/O, while waiting for commitment.
"fsync" is really a clunky way of getting that second completion.
So all that's needed is encapsulating it in a library that provides those different abstractions.
Well ok, we could also use an additional flag for linkat() that allows atomically replacing the target to close that tiny window where a temporary file might get left behind after a crash.
- Unit files: memfd, fdatasync, linkat, fsync(dir)
- Temporary files. memfd, preferably within /tmp to avoid any disk I/O unless you have a particularly large file to buffer
- Log files. This is trickier. Probably requires running on a fs that supports reflinks and creating a new memfd that’s a true clone of the original file, appending, & then replacing it. The one piece that is incompatible with this is that other file descriptors are stale and you have to reopen to see new data but that’s at odds with APIs available today - either readers can see partial writes or you have to have a new file. There have been proposals to allow buffering writes but delaying publishing the metadata acking the data but that hasn’t gone anywhere unfortunately.
- Managed files. I’m not sure that completions are the right contract here because technically the kernel could defer that completion indefinitely whereas databases typical need some kind of latency bounds of “OK - I really want this to commit to disk nowish”.
memfd is for files in RAM. linkat cannot get them onto disk. What you seem to have in mind is what the grandparent post already described: "O_TMPFILE + linkat".
Durability is totally different from concurrency or multiprocess consistency.
There are already guarantees about how we can order reads and writes in a shared file. Very well defined guarantees -- ones we've built entire RDBMS systems on top of.
Guaranteeing durability (persistence across a crash) is a totally separate problem from understanding write visibility between threads.
Ah, the SQLite people. I was thinking in terms of databases such as MySQL and Postgres where one process owns the files and client programs communicate with it.
Syncing data to durable storage is totally unrelated to visibility guarantees between multiple threads or processes working within the same file (be it using write, mmap, whatever)
The visibility issues you're raising, basically implementing views and transactions, would not be applicable to any of these systems.
You're confusing durability guarantees (fsync, msync) with visibility guarantees -- the latter of which are just standard memory access issues.
Is it though? I think single writer+many readers is a very common use case. For example I want a compiler to read a file I have open in my IDE. Or I want to be able to open a log file while the application is still running.
Atomicity is orthogonal to durability. rename() is atomic and always has been atomic. It is atomic without guaranteeing durability.
In fact, to guarantee durability after rename you must:
1) rename(a, b)
2) open(the parent dir)
3) fsync(the parent dir)
This is to guarantee durability of the metadata written by rename() in the dirent.
All of this is totally separate from guaranteeing the durability of the data in the file, which requires its own separate fsync() on the file itself.
Atomicity and durability are never the same. This article is gibberish.
We say that rename() is atomic because you will either see file A or file B. There are no other possible states.
With a crash, writes might not be committed. You might get an earlier filesystem state.
And the whole point of TFA is that this is perfectly fine as long as the metadata writes aren't committed before the data writes. That is:
Precondition: File at path A exits with data X
1. Create file at path B
2. Write data Y to file B.
3. Close file B
4. Rename B to A
5. Crash
---
As long as the contents in the file at path A are either X or Y, then we have achieve atomicity in the sense used in TFA. This is what we mean by rename acting "atomically across crashes"
Note that we only care about the file at path A; all of these are fine:
- File at path B exists and has data Y
- File at path B exists and has garbage data
- File at path B doesn't exist
- File at path B exists and is of size 0
Given a link to a file at path A exists. The file contains data X. There may be other links at paths C, D and E.
1. Acquire a filesystem-global lock on link ops (link/unlink et al, but not read/write).
2. Create a link to X at path B
3. Remove the link at path A
4. Release the filesystem-global lock.
Note: The data in X is not relevant. The data in X may be undergoing active modification while the above 4 steps are performed. The data in X may be memory mapped and may change many times during the "atomic" rename() syscall.
The atomic properties of rename() are only vis a vis link/unlink semantics within a single filesystem. Other files can't be created or deleted while the rename() is in progress. File data however is under no such restriction -- and in fact it cannot be because the file may be memory-mapped and modified without any syscall interactions. There is no system call sequence point to gate reads and writes.
What you are describing is fundamentally incompatible with how the system actually works. What you describe cannot be made to work, ever. It is architecturally invalid at a fundamental level.
> File data however is under no such restriction -- and in fact it cannot be because the file may be memory-mapped and modified without any syscall interactions. There is no system call sequence point to gate reads and writes.
Note a complete lack of mmap() in the list of operations above; you are trying to trust what I am asking for into something that is impossible, when what I'm asking for is definitely possible.
[Edit]
I'm also well aware of how rename works; the ask is that the link is not committed before the data. This is possible to do at the expense of some performance, but possibly less performance then using existing commit primitives (e.g. fsync or fdatasync)
Because it is not an issue of atomic behavior but merely ordering there's no reason to modify the kernel or touch syscalls. It would be far more reliable to provide this in a library -- rename(3) -- which would then automatically provide the benefit on every posix compatible filesystem.
It would also allow for a use case of wanting to rename a file without forcing a full data sync -- which could be performance critical behavior in some scenarios.
To recap:
No filesystems provide atomic syncing of data during rename(). The words don't even make sense. Ext doesn't do it, zfs doesn't do it, xfs doesn't do it. Achieving this would require mechanisms that don't exist.
Some filesystems do guarantee that an fsync will occur alongside a rename. There is no specific guarantee as to what file state will be synced. These filesystems do this to mask the impact of folks mis-using the API, paying an efficiency cost as a result.
In ext4, for example, the sync does NOT occur within the same transaction as the rename. The sync is NOT atomic (it would be wild if this were the case, as noted above -- architecturally inconsistent and extremely non-performant)
It would be ideal for this to happen in libc rather than in the kernel.
Yes everyone in the discussion agrees on this fact
> ...but it is not atomic, merely ordered. You've switched from talking about one to talking about the other and they are not the same
If the operations we are talking about are ordered, then this provides the atomicity that TFA is asking for. Properly ordering operations is a very common way of making a series of steps atomic.
No, it doesn't. I think we're reaching the heart of the disagreement here - around what "atomic" means.
Atomic means indivisible. It means that other things cannot happen at the same time - that the individual (possibly ordered) steps of an operation cannot be observed while in progress.
This is why it's absolutely nonsense to talk about atomicity across a crash.
The fundamental design of unix files and memory preclude this. It can't be done.
> ...the operation we wish to be atomic is 'replacing the mapping path A->data X with path A->data Y'.
The ordering makes this operation atomic! The fact that it might leave another file around (if a crash happens) is okay.
Again, manipulating links has nothing to do with file data.
You can sync to durable storage at any point, yes. But you cannot do it in a semantically useful fashion.
"I don't think anyone is talking about situations where B is being actively written to while the rename happens."
Well, you specifically are ignoring this. I'm not.
Filesystems are free to generate extra syncs whenever they like. You could even have a filesystem sync every single write operation to durable storage - why not?
I think this would be a benefit, because first you don't need userspace developers dig into the gory details of dirent metadata durability, and second because by making this explicit it helps put pressure on filesystem developers to optimize what the users actually want.
"I think this would be a benefit, because first you don't need userspace developers dig into the gory details of dirent metadata durability"
This is what libraries are for, not syscalls.
The problem here is you're mixing up two unrelated concepts: Durability and atomicity.
Provide new API calls with precisely defined semantics.
Rather have this rigamarole with fsync and rename, provide an actual syscall with the actual effect the userspace developers are looking for. Eg, an atomic_replace() syscall that ensures that either a file is replaced with a new, fully written to disk version, or nothing happens.
The main problem I see is that this of course would be Linux specific, so of course somebody would build a library to either invoke the syscall or do the fsync/rename mess underneath, and this would of course run into the same exact problem on those systems.