News for hackers. Stuff that matters.
Re:Freqfs: In-memory filesystem cache for Rust by haydnv on Friday October 01, 2021 @02:56PM (#28722166)
One advantage is consistency across host platforms, but the main advantage is that the file data can be accessed (and mutated) in memory in a deserialized format. If you let the OS take care of it, you would still have the overhead of serializing & deserializing a file every time it's accessed.
Re:Freqfs: In-memory filesystem cache for Rust by dfranke on Friday October 01, 2021 @03:16PM (#28722343)
That's what mmap is for.
Re:Freqfs: In-memory filesystem cache for Rust by haydnv on Friday October 01, 2021 @04:01PM (#28722773)
It might be possible to replace freqfs with mmap on a POSIX OS, but a) you would still have to implement your own read-write lock, and b) you would (I think probably?) lose some consistency in behavior across different host operating systems.
Re:Freqfs: In-memory filesystem cache for Rust by vlovich123 on Friday October 01, 2021 @04:15PM (#28722949)
Which OSes does this run on that doesn’t have some kind of mmap operation?
Re:Freqfs: In-memory filesystem cache for Rust by haydnv on Friday October 01, 2021 @05:17PM (#28723467)
It should work on Windows (because tokio::fs works on Windows) although I have not personally tested this
Re:Freqfs: In-memory filesystem cache for Rust by julian37 on Friday October 01, 2021 @05:35PM (#28723643)
You can do mmap on Windows, eg. https://github.com/danburkert/memmap-rs
Re:Freqfs: In-memory filesystem cache for Rust by gpderetta on Friday October 01, 2021 @04:10PM (#28722895)
mmaps for read, explicit API for writing, a-la LMDB. Buggy readers can read inconsistent data but cannot corrupt the os.
Re:Freqfs: In-memory filesystem cache for Rust by otterley on Friday October 01, 2021 @04:39PM (#28723181)
Corrupt the OS? How might that happen?
Re:Freqfs: In-memory filesystem cache for Rust by gpderetta on Friday October 01, 2021 @06:30PM (#28724066)
Sorry, I meant the DB!
Re:Freqfs: In-memory filesystem cache for Rust by alexruf on Friday October 01, 2021 @03:23PM (#28722419)
Personally I don’t see a scenario for myself, but I can imagine that there are some where this might be useful. But isn’t there a extremely high risk of data loss an inconsistency when adding an extra layer on top of OS file system handling?
Re:Freqfs: In-memory filesystem cache for Rust by ericbarrett on Friday October 01, 2021 @04:35PM (#28723144)
Freqfs seems like a shim you'd add to an existing project for a quick optimization. Whereas mmap et al. are "better" the same way any specific, built-to-purpose code will be "better" than just bolting a framework on. Sometimes it's the right call to do the extra work; sometimes it's 100% more effort (both development and maintenance) for an extra 10% gain.
Re:Freqfs: In-memory filesystem cache for Rust by haydnv on Friday October 01, 2021 @04:04PM (#28722802)
If there is any concurrent access to cached files not through freqfs, there is a risk of inconsistency and crashes.