Spotting and Avoiding Heap Fragmentation in Rust Applications

Re:Spotting and Avoiding Heap Fragmentation in Rust Applicat... by dgacmu on Thursday April 06, 2023 @12:58PM (#35471179)

No - heap fragmentation actually leaves memory unusable by the workload. It may or may not slow down allocation depending on the design of the allocator.

(The analogy to file system fragmentation for memory is that when physical pages are allocated in a discontinuous manner, it prevents some optimizations like coalescing them into hugepages, which for some workloads can help with TLB hit rate.)

[ Parent | on HN ]

Re:Spotting and Avoiding Heap Fragmentation in Rust Applicat... by yakubin on Thursday April 06, 2023 @02:45PM (#35472613)

Filesystem may also run out of inodes, even though there is still plenty of space.

[ Parent | on HN ]

Re:Spotting and Avoiding Heap Fragmentation in Rust Applicat... by wongarsu on Thursday April 06, 2023 @03:41PM (#35473359)

I don't think either FAT32 or NTFS suffer this problem, they allow metadata to become fragmented instead. Certainly a different design philosophy to the unixy file systems.

[ Parent | on HN ]

Re:Spotting and Avoiding Heap Fragmentation in Rust Applicat... by magicalhippo on Thursday April 06, 2023 @06:03PM (#35475154)

NTFS most definitely suffers from this[1]. We ran into it with a customer who had a particularly fragmented database file.

IIRC it's due to a combination of relying on fixed-sized "pages" to hold fragment pointers and a limited number of page indirections. That is the root page can point to a sub-pages, which again can point to sub-pages, but those sub-sub-pages have to point to the actual fragments. Or something along those lines.

[1]: https://support.microsoft.com/en-au/topic/a-heavily-fragment...

[ Parent | on HN ]

Re:Spotting and Avoiding Heap Fragmentation in Rust Applicat... by cryptonector on Thursday April 06, 2023 @05:24PM (#35474702)

Or you should use ZFS and not worry about that.

[ Parent | on HN ]