This is the fundamental problem: When you allow the OS (or the compiler, or the CPU) to re-order operations in the name of efficiency you lose control over intermediate states, and so you cannot guarantee that these intermediates states are consistent with respect to the semantics that you care about. And this is the even more fundamental problem: our entire programming methodology has revolved around describing what we want the computer to do rather than what we want to to achieve. Optimizations then have to reverse-engineer our instructions and make their best guesses as to what we really meant (e.g. "This is dead code. It cannot possibly affect the end result. Therefore it can be safely eliminated.) Sometimes (often?) those guesses are wrong. When they are, we typically only find out about it after the mismatch between our expectations and the computer's have manifested themselves in some undesirable (and often unrecoverable) way.
No, it can work, provided that the temporary file is fsynced before being renamed, the parent directory is fsynced after renaming the file, and that the application only considers the rename to have taken place after the parent directory is fsynced (not after the rename call itself).
[0] Here's follow(1) in my homemade PL:
#!/usr/bin/env imp
(when (not-eq? (length script-args) 1)
(write (sprintln "usage: " script-name " <path>") stderr)
(exit 1))
(let path (car script-args)
f (open path)
((proc (stat-then)
(let stat-now (stat f)
(seq (when (and? stat-then
(lt? (find 'size stat-now)
(find 'size stat-then)))
# The file was truncated. Read from the start.
(seek s-set 0 f))
(let bytes (read default-chunk f)
(seq (when bytes (print bytes))
(sleep 1000)
(recur stat-now))))))
FALSE))if you have long lived sessions tail'ing logs that get rotated (or truncated), forgetting -F is a good way to eventually end up confused, ctrl-c'ing, and finally cursing.
* Single file + log: fine grained locking in a shared C data structure. Yuck!
* Write new then move: transactional reference to a shared storage location, something like Clojure's refs. Easy enough.
The latter clearly provides the properties we'd like, the former may but it's a lot more complicated to verify and there are tons of corner cases. So I think move new file over old file is the simpler strategy and way easier to reason about.
The obvious downside is that this temporarily uses twice the size of the dataset. However, that is usually mitigated by splitting the data into multiple files, and/or applying this only to applications that don't need to store gigabytes in the first place.
Clojure's approach again provides an interesting solution to saving space. Taking the idea of splitting data into multiple files to the logical conclusion, you end up with the structure sharing used in efficient immutable data structures.
Linked paper mention the many popular programs forgets about (1).
1. Write changed the data into a temporary file in the same directory (don't touch the original file)
2. Move new file over old file
Does this lead to a simpler strategy that is easier to reason about, where it is less likely for programmer to get it wrong? At least I see this strategy being applied more often than the "single file + log" approach.
The obvious downside is that this temporarily uses twice the size of the dataset. However, that is usually mitigated by splitting the data into multiple files, and/or applying this only to applications that don't need to store gigabytes in the first place.