News for hackers. Stuff that matters.
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by robertnishihara on Wednesday July 31, 2024 @07:03PM (#41124465)
Yeah, mmap, I think this is the relevant line [1].

Fun fact, very early on, we used to create one mmapped file per serialized object, but that very quickly broke down.

Then we switched to mmapping one large file at the start and storing all of the serialized objects in that file. But then as objects get allocated and deallocated, you need to manage the memory inside of that mmapped file, and we just repurposed a malloc implementation to handle that.

[1] https://github.com/ray-project/ray/blob/21202f6ddc3ceaf74fbc...

There are no comments on this comment yet.