News for hackers. Stuff that matters.

A comment on A way to do atomic writes.

Re:A way to do atomic writes by ww520 on Wednesday May 29, 2019 @03:34PM (#20044393)
Appending log based file system has a lot of appeals. A lot of hard problems, like atomic write, become trivial. A log based FS shares similar properties as the transaction log in RDBMS, making consistency and recovery easy. Write performance is fantastic. With enough cache memory, read performance should be good, too. The only down-size is the performance hit when doing garbage collection. More research in this area would be helpful.
[ Parent | on HN ]
Re:A way to do atomic writes by burmecia on Thursday May 30, 2019 @12:18AM (#20048346)
Totally agree. Log based file system makes things much easier, such as atomic write, data consistency and failure recovery. Along with COW semantics, transaction isolation also becomes easier. By its append-only writing nature, file versioning is trivial as well. Garbage collection is performance downside indeed, but it can be mitigated by COW switching and delayed bulk collection IMHO. Overall, the cost is worthwhile and that is what I am currently doing in ZboxFS: https://github.com/zboxfs/zbox.
[ Parent | on HN ]
Re:A way to do atomic writes by ww520 on Thursday May 30, 2019 @07:24PM (#20056953)
That looks promising. Worth taking a look.
[ Parent | on HN ]
Re:A way to do atomic writes by ww520 on Wednesday May 29, 2019 @09:49PM (#20047622)
Journaling FS only puts metadata and FS commands in log, not structuring the FS itself as an appending log.
[ Parent | on HN ]
Re:A way to do atomic writes by Scriptor on Wednesday May 29, 2019 @05:04PM (#20045468)
Haven't log-based file systems been a thing for a while now? https://en.wikipedia.org/wiki/Log-structured_file_system
[ Parent | on HN ]
Re:A way to do atomic writes by notacoward on Wednesday May 29, 2019 @07:25PM (#20046789)
They have, but they had fallen way out of favor for a couple of reasons - write amplification, cost/complexity of segment cleaning, poor performance for some workloads, etc. They've experienced a bit of a resurgence thanks to flash and SMR, though. Personally, I've always thought log-structured filesystems were elegant and wish they'd been more actively developed during the long "winter" during which their COW/write-anywhere cousins stole all the limelight.
[ Parent | on HN ]
Re:A way to do atomic writes by ww520 on Wednesday May 29, 2019 @10:43PM (#20047896)
Was hoping there are more research attempt at the Log Structured FS. For example,

- Borrow some ideas from generational garbage collection. Young generation in SSD (or mirrored in RAM) with copying GC to get rid of old versions of fast changing data blocks.

- Utilize some deduplication techniques with content based signature.

[ Parent | on HN ]
Re:A way to do atomic writes by kjeetgill on Thursday May 30, 2019 @01:35AM (#20048688)
I think elements of that generationality are the foundations of the Log Structured Merge Trees used by KV Stores like LevelDB and RocksDB. Atleast I think that's the same concept, I'm not well read on filesystems.
[ Parent | on HN ]
Re:A way to do atomic writes by ww520 on Thursday May 30, 2019 @07:59PM (#20057102)
Yes, LSMT is a good example of pushing the idea of a hybrid append log and in memory data structure further.

However, LSMT is for relatively smaller data set, i.e. ordered key-value. It has worse write amplification than a simple append log. The level-0 memtable flushed to the write-ahead-log counts as one write. Writing to the level-1 sorted files counts as 2nd write. Merging the sorted files counts as 3rd write. There're 2~3 writes per change.

Also it doesn't offer help to address the frequent update block problem. All versions of a data change are written to disk. A merge is needed to get rid of the old versions.

But it has a number of good implementation ideas that can be borrowed.

[ Parent | on HN ]