Re:Writing a Memory Allocator for Fast Serialization
by bertr4nd on Monday July 03, 2017 @09:05PM
(#14692728)
What happens if the program unexpectedly terminates during an update to these persistent data structures? Is the mmap'ed file corrupted, or still in a usable state?
[ Parent
| on HN ]
Re:Writing a Memory Allocator for Fast Serialization
by dryman on Monday July 03, 2017 @10:15PM
(#14693011)
I don't have a good answer to this question yet. For now I only create the heap in swap, write it to disk, and then use it as read only mmap. To ensure the written file is valid, one can write it to a temporal file first, once confirmed the file is written, then mv the file to desired file name and location. This works for immutable data, but is a big blocker for me to make OPIC work on mutable back store.
This problem is generally hard. See [Ensuring data reaches disk](https://lwn.net/Articles/457667/)
Re:Writing a Memory Allocator for Fast Serialization
by hyc_symas on Tuesday July 04, 2017 @08:07AM
(#14695339)
You should have looked more closely at LMDB, which already solves this problem. Also you could look at LMDB's API for using fixed-address mmap, which lets you store pointer-based structures without any deserialization step at all.
Re:Writing a Memory Allocator for Fast Serialization
by pjscott on Monday July 03, 2017 @09:44PM
(#14692895)
The operating system will dutifully write back to disk whatever bytes your program left on those memory-mapped pages at the moment it terminated. If your data structures were in an inconsistent state -- pointers to nowhere, say -- then your file will get in the same inconsistent state, i.e. corrupted.