Re:Memray: a memory profiler for Python
by mjsir911 on Wednesday April 20, 2022 @04:56PM
(#31102891)
I've had good luck with yappi for debugging asynchronous code, not really focused on memory usage but walltime & such.
Re:Memray: a memory profiler for Python
by uniqueuid on Wednesday April 20, 2022 @05:02PM
(#31102965)
Thanks, that looks useful!
Re:Memray: a memory profiler for Python
by kungfufrog on Wednesday April 20, 2022 @06:56PM
(#31104122)
https://github.com/plasma-umass/scalene check out scalene!
+1 for scalene! I love how it breaks down consumption line-by-line, and it's worked very well in my async/multithreaded usecase
Re:Memray: a memory profiler for Python
by erwincoumans on Wednesday April 20, 2022 @05:36PM
(#31103361)
I like open source Perfetto UI (formerly Google Chrome about::/tracing)
Wrote my own timing trace json export for Python and C++, across threads and processes. The docs have some pointers to creating your own traces with the Tracing SDK.
Re:Memray: a memory profiler for Python
by knlb2022 on Wednesday April 20, 2022 @06:09PM
(#31103726)
I built https://github.com/kunalb/panopticon to export perfetto/chrome compatible traces and also draw arrows between async functions. (I think the arrows are only supported in about://tracing though).
Looks like you can use magic-trace with a perfetto based UI. https://github.com/janestreet/magic-trace
viztracer can create traces for python programs. Not sure about async-io awareness though.
Actually it has explicit support for async task based reporting:
Re:Memray: a memory profiler for Python
by zackangelo on Wednesday April 20, 2022 @04:57PM
(#31102902)
Do you mean specifically for Python? There's this for async Rust:
Re:Memray: a memory profiler for Python
by p403n1x87 on Thursday April 21, 2022 @09:44AM
(#31109285)
This should work with anything https://github.com/P403n1x87/austin
My experience has been that the concurrent nature of coroutines can make it hard to reason about what's going on at a particular point in time. If you don't know how many things you're awaiting on at a specific moment (and what potential external stuff they may be interacting with), it's not exactly easy to identify memory usage of codepaths.