Re:Memory Management in Python
by shakna on Monday August 05, 2019 @08:14AM
(#20613448)
This has had me thinking about the Python/malloc story for a little bit.
It does make sense CPython is tuned more towards runtime than startup time. One is a continual cost, one is paid once and doesn't matter for a large variety of workloads.
But if CPython really is make 1000+ allocations during startup, then it is probably going to fragment badly, which slow down everything, and into the future as well...
Seems that's not a crazy idea. [0] Zapier linked CPython to jemalloc which tends to fragment less and saw 40% memory gains.
I guess if you took the time to tune jemalloc to your workload you should be able to see significant performance improvements at startup as well as in the hot path.
It does make sense CPython is tuned more towards runtime than startup time. One is a continual cost, one is paid once and doesn't matter for a large variety of workloads.
But if CPython really is make 1000+ allocations during startup, then it is probably going to fragment badly, which slow down everything, and into the future as well...
Seems that's not a crazy idea. [0] Zapier linked CPython to jemalloc which tends to fragment less and saw 40% memory gains.
I guess if you took the time to tune jemalloc to your workload you should be able to see significant performance improvements at startup as well as in the hot path.
[0] https://zapier.com/engineering/celery-python-jemalloc/