This has had me thinking about the Python/malloc story for a little bit.

It does make sense CPython is tuned more towards runtime than startup time. One is a continual cost, one is paid once and doesn't matter for a large variety of workloads.

But if CPython really is make 1000+ allocations during startup, then it is probably going to fragment badly, which slow down everything, and into the future as well...

Seems that's not a crazy idea. [0] Zapier linked CPython to jemalloc which tends to fragment less and saw 40% memory gains.

I guess if you took the time to tune jemalloc to your workload you should be able to see significant performance improvements at startup as well as in the hot path.

[0] https://zapier.com/engineering/celery-python-jemalloc/

With it being a very popular python implementation, one wonders why such optimizations haven't been implemented yet. 60% sounds huge.

Probably because overall Python startup time is still dominated by site.py/pkg_resources spidering the filesystem and importing half the world; as long as that exists for most users, it's hard to be motivated to make speedups anywhere else.

cf. https://github.com/pypa/setuptools/issues/510

The problem is so bad that it's one of the top quoted reasons that hg is considering a gradual rewrite to rust:

https://www.mercurial-scm.org/wiki/OxidationPlan

Hilariously, my pentadactyl shortcut for the oxidation plan was HG. I took that as a sign to read the article. Has hg ever considered a long-running "daemon-like" process instead? It could be instantiated automatically by the first call to hg and then it can die after it hasn't been used for a while. It would probably be racy and hard to support on multiple OS's though...

They do kind of have this: https://www.mercurial-scm.org/wiki/CommandServer

Though it looks like it's more about supplying a way to embed hq in a proprietary (or non-Python) application via subprocess communication rather than directly importing the GPL modules or having to string-parse responses. There doesn't seem to be provision for interacting with the server directly from the command line, which is odd, since this pattern is well established from Java land (bazel, for example).

The pkg_resources problem is specific for programs that use entry points (and it can be avoided), not a general CPython issue, and AFAICT is not related to Mercurial's plan to include Rust.

Interesting; I hadn't realised this, thanks for the clarification.

I'm aware of yapsy as a lightweight plugin system, but entry_points really is the gold standard in terms of Python extensibility. It's a shame it can't be made more performant.

Thanks for this, it makes for an interesting read, especially for someone who only dips his toe in python programming lightly once in a while.