Re:Zero-copy deserialization in Julia
by rwmj on Sunday September 02, 2018 @08:17AM
(#17896717)
This reminds me of the OCaml "Ancient" library[1] that I wrote. It lets you have an extra heap of OCaml objects which can be stored in an mmap'd file and even shared between processes (although the sharing must be read-only and has a number of problems like everything has to be mapped at the same address which conflicts with ASLR - we didn't use pointer<->offset conversion as done in this article because it would touch every page and you'd end up with no sharing).
I originally wrote it to analyze large data sets (where "large" meant 16+ GB which was much larger than the commonly available RAM at the time).
It integrated nicely with OCaml. You could create ordinary OCaml objects then incrementally "mark" them so those objects would be moved (recursively) into the ancient heap. The objects could still be accessed as if they were ordinary OCaml objects even if they were on the ancient heap. On the other side you could mmap a previous heap and access the objects as if they were regular OCaml objects directly (with a few shortcomings - see README). Depending on your access pattern this worked well even if available RAM was much smaller than the size of the data set.
Re:Zero-copy deserialization in Julia
by jamii on Sunday September 02, 2018 @08:22AM
(#17896732)
I used Ancient years ago for https://github.com/jamii/texsearch to stop the GC pauses caused by pointlessly traversing the huge immutable index. I don't know how else I would have met the latency requirements. Thanks for writing it :)
I originally wrote it to analyze large data sets (where "large" meant 16+ GB which was much larger than the commonly available RAM at the time).
It integrated nicely with OCaml. You could create ordinary OCaml objects then incrementally "mark" them so those objects would be moved (recursively) into the ancient heap. The objects could still be accessed as if they were ordinary OCaml objects even if they were on the ancient heap. On the other side you could mmap a previous heap and access the objects as if they were regular OCaml objects directly (with a few shortcomings - see README). Depending on your access pattern this worked well even if available RAM was much smaller than the size of the data set.
[1] http://git.annexia.org/?p=ocaml-ancient.git;a=blob;f=README....