haydnv

mmap does map file contents into memory but does not provide read-write locking. The main limitation is that a file is not necessarily the same thing as the Rust data structure that it encodes. For example, if you store a Rust Vec in a file, loading the file with mmap won't allow you to `push` a new item onto the Vec.

I realized based on several of the comments here that I should have included a comparison with OS filesystem caching in the documentation for freqfs. I will update this in the next release.

The major advantage of freqfs over just letting the OS handle file caching is that with freqfs you can read and mutate the data that your file represents purely in memory. For example, if you implement a BTree node as a struct, you can just borrow the struct mutably and update it, and it will only be synchronized with the filesystem in the event that it's evicted (or you explicitly call `sync`). This avoids a lot of (de)serialization overhead and defensive coding against an out-of-memory error.

Again, I will update the documentation to clarify.

I think if you call your precompression function in FileLoad::load it should do what you need--please file an issue if this is not the case: https://github.com/haydnv/freqfs/issues

I don't know that it's fair to say it's "doubling" the memory use of each file because the OS cache memory is still "free" from the perspective of an application. Where it comes in handy is an applications like databases or training an ML model where there are hot spots that get accessed/updated extremely frequently--then the application doesn't have to incur serialization overhead in order to read/write the data that the file encodes (although as another poster pointed out it might also be possible to do this with mmap).

Yeah I just learned about tokio-uring and I'm planning to get it into the next major release of freqfs

That's great! Please let me know how it goes! Feel free to file any bug reports or feature requests here: https://github.com/haydnv/freqfs/issues

It should work on Windows (because tokio::fs works on Windows) although I have not personally tested this

If there is any concurrent access to cached files not through freqfs, there is a risk of inconsistency and crashes.

It might be possible to replace freqfs with mmap on a POSIX OS, but a) you would still have to implement your own read-write lock, and b) you would (I think probably?) lose some consistency in behavior across different host operating systems.

One advantage is consistency across host platforms, but the main advantage is that the file data can be accessed (and mutated) in memory in a deserialized format. If you let the OS take care of it, you would still have the overhead of serializing & deserializing a file every time it's accessed.

I think what's really interesting about this that most readers seem to overlook is the claims about subluminal warp drive, which is practically realizable in a way that the science-fictional faster-than-light case is not