Re:Memory Profiling Part 1. Introduction
by kouteiheika on Thursday February 15, 2024 @01:30AM
(#39379746)
> Instead I've added random sampling - e.g. if ptr % modulo > level - output it or not.
A good alternative that also cuts down on the amount of data significantly is to strip away temporary allocations, say, only emit those allocations which were alive for at least X seconds.
Re:Memory Profiling Part 1. Introduction
by malkia on Thursday February 15, 2024 @01:32PM
(#39386409)
Ah good idea! It kind of calls for out-of-process to filter this out, or post-process (maybe there is common tooling around).
Re:Memory Profiling Part 1. Introduction
by milianw on Thursday February 15, 2024 @05:23AM
(#39381034)
Heaptrack can handle millions of allocations per second, and thanks to the deduplication built into the trace file format combined with zstd compression, the overhead is pretty manageable. It's pretty hard to get "hundreds of GB" of data recorded.
Another thing: you can always just runtime attach and profile a partial time to reduce the amount of data recorded.
Finally: if your tools do so many allocations, maybe you should consider optimizing them...
Re:Memory Profiling Part 1. Introduction
by malkia on Thursday February 15, 2024 @01:32PM
(#39386418)
Thanks! Should look into heaptrack - reading that it's linux only, but probably can be adopted for Windows?
Re:Memory Profiling Part 1. Introduction
by milianw on Monday February 19, 2024 @09:34AM
(#39430134)
MTuner exists for Windows which is similar but different. Porting heaptrack is not straight forward, as it heavily relies on libraries that do not work on Windows or do not support Windows executables and debugging information, such as libunwind and elfutils.
Instead I've added random sampling - e.g. if ptr % modulo > level - output it or not.
Another factor that slows down is doing a callstack capture. It's not for free at all, like on Windows it has to go through the exception handlers, etc. I think perfetto simply captures the whole stack (need to check again), and then offline decodes it - or something like this.
Windows ETW Tracing can also capture the stack, but I guess it'll incur also some penalty - it can't come for free.
I also wish there was some kind of standard binary format for emitting alloc/free sequences with callstack/etc.