News for hackers. Stuff that matters.
Re:PostgreSQL's fsync surprise by gruez on Wednesday May 02, 2018 @10:55PM (#16983033)
>Ideally I want:

everything you describe will murder IO performance because you're essentially implementing a database with MVCC. it makes much more sense to implement what you described as a VFS library.

Re:PostgreSQL's fsync surprise by sethhochberg on Wednesday May 02, 2018 @11:12PM (#16983088)
ZFS gets a lot of the way there - and indeed has lots of similarities to a database in its internals.

(In fact, ZFS with `sync=always` set on the dataset would do exactly what the parent commenter is asking for without requiring a whole new API. If someone really cared about performance they could even make this perform well with a battery-backed DRAM device for their intent log... but by that point we're way outside of the discussion of a general-purpose filesystem. If you really need guaranteed consistency for every write _without_ crippling performance, ZFS can do it for you at a hardware cost. Writes get cached in RAM, batched, flushed in transaction groups, and the fast/durable SLOG protects against corruption or loss of power or device failure without harming performance much. $$$ though. You'd better have quite a bit of RAM, and need a proper enterprise-grade DRAM storage device - or probably two in a mirror if your data is this critical).

Re:PostgreSQL's fsync surprise by josephg on Wednesday May 02, 2018 @11:26PM (#16983139)
Murder the IO performance of what, exactly? For what use case is fast non-atomic writes something we care about? Atomic-append is the IO operation we should be optimizing for! What murders IO performance is trying to reimplement a second write-ahead-log in userspace on top of fsync. Which is something modern databases have to actually do in order to meet their correctness requirements. This complexity and performance cost would be much lower with knowledge of the kernel & control over filesystem internals.

And for every case of something that would have lower IO performance with atomic appends, I'm sure we can think of ways the kernel could make those atomic writes fast. Lets say you're copying a big file by wrapping a transaction around a read();write() loop. The OS could make the transaction extremely cheap by simply not changing the file's INODE list (or whatever) until the write has completed. In a filesystem like ZFS, using a transaction for this would let ZFS not update its filesystem checksums until the copy has finished, which would probably in aggregate result in improved performance.