Try "python -vv -c 'pass'" - I'm only showing the first few dozen lines, and I've trimmed some of the paths for conciseness:
% python -vv -c 'pass'
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# miniconda3/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/__init__.py
# code object from 'miniconda3/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc'
# trying miniconda3/lib/python3.7/codecs.cpython-37m-darwin.so
# trying miniconda3/lib/python3.7/codecs.abi3.so
# trying miniconda3/lib/python3.7/codecs.so
# trying miniconda3/lib/python3.7/codecs.py
# miniconda3/lib/python3.7/__pycache__/codecs.cpython-37.pyc matches miniconda3/lib/python3.7/codecs.py
# code object from 'miniconda3/lib/python3.7/__pycache__/codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd44c90>
# trying miniconda3/lib/python3.7/encodings/aliases.cpython-37m-darwin.so
# trying miniconda3/lib/python3.7/encodings/aliases.abi3.so
# trying miniconda3/lib/python3.7/encodings/aliases.so
# trying miniconda3/lib/python3.7/encodings/aliases.py
# miniconda3/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/aliases.py
# code object from 'miniconda3/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd67d10>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd440d0>
# trying miniconda3/lib/python3.7/encodings/utf_8.cpython-37m-darwin.so
# trying miniconda3/lib/python3.7/encodings/utf_8.abi3.so
# trying miniconda3/lib/python3.7/encodings/utf_8.so
# trying miniconda3/lib/python3.7/encodings/utf_8.py
# miniconda3/lib/python3.7/encodings/__pycache__/utf_8.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/utf_8.py
# code object from 'miniconda3/lib/python3.7/encodings/__pycache__/utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd44bd0>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# trying miniconda3/lib/python3.7/encodings/latin_1.cpython-37m-darwin.so
# trying miniconda3/lib/python3.7/encodings/latin_1.abi3.so
# trying miniconda3/lib/python3.7/encodings/latin_1.so
# trying miniconda3/lib/python3.7/encodings/latin_1.py
# miniconda3/lib/python3.7/encodings/__pycache__/latin_1.cpython-37.pyc
matches miniconda3/lib/python3.7/encodings/latin_1
... many, many more lines omitted ...
This can be sped up a lot using a zipimport of the Python standard library, https://docs.python.org/3/library/zipimport.html?highlight=z... , where all of the standard library is put in a zipfile. Then there's a file access to get the zip metadata.One of the things that bugged me in Python2 was that every startup imported UserDict:
# trying python2.7/UserDict.so
# trying python2.7/UserDictmodule.so
# trying python2.7/UserDict.py
# python2.7/UserDict.pyc matches python2.7/UserDict.py
import UserDict # precompiled from python2.7/UserDict.pyc
This is because os.environ was an instance of UserDict: % python2.7 -c 'import os; print(os.environ.__class__.__bases__)'
(<class UserDict.IterableUserDict at 0x1029b14c8>,)
Under Python3 this is spelled collections.abc.MutableMapping: % python3 -c 'import os; print(os.environ.__class__.__bases__)'
(<class 'collections.abc.MutableMapping'>,)
which triggers its own set of imports: # trying python3.6/collections/abc.cpython-36m-darwin.so
# trying python3.6/collections/abc.abi3.so
# trying python3.6/collections/abc.so
# trying python3.6/collections/abc.py
# python3.6/collections/__pycache__/abc.cpython-36.pyc matches python3.6/collections/abc.py
# code object from 'python3.6/collections/__pycache__/abc.cpython-36.pyc'
import 'collections.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x103c2ecf8>
There's better performance using SDD than a HDD, which is in turn better than using a networked filesystem.It's just hard to fix.
I'm not sure the os.environ example I gave is low-hanging fruit now. The collections.abc module might be imported anyway.
This is neat! Python 3.7 added the `PYTHONPROFILEIMPORTTIME=1` environment variable to help track down these sorts of import overheads:
% env PYTHONPROFILEIMPORTTIME=1 python -c pass
import time: self [us] | cumulative | imported package
import time: 523 | 523 | zipimport
import time: 722 | 722 | _frozen_importlib_external
import time: 156 | 156 | _codecs
import time: 2254 | 2409 | codecs
import time: 1293 | 1293 | encodings.aliases
import time: 7192 | 10893 | encodings
import time: 1108 | 1108 | encodings.utf_8
import time: 182 | 182 | _signal
import time: 1069 | 1069 | encodings.latin_1
import time: 395 | 395 | _abc
import time: 1486 | 1881 | abc
import time: 1540 | 3420 | io
import time: 100 | 100 | _stat
import time: 975 | 1075 | stat
import time: 1481 | 1481 | genericpath
import time: 1734 | 3214 | posixpath
import time: 2558 | 2558 | _collections_abc
import time: 2234 | 9079 | os
import time: 1407 | 1407 | _sitebuiltins
import time: 3498 | 3498 | sitecustomize
import time: 85 | 85 | usercustomize
import time: 4129 | 18196 | site
Investigating further, the "import os" which triggered the UserDict/collections.abc is a consequence of "import site". If I use "python -S" then those aren't imported.For some of my programs, Python startup time is the main overhead. I avoid NumPy and SciPy if at all possible because they have a huge startup overhead.
Some of this is inherent in those packages. NumPy internally imports everything so someone can do "import numpy as np; np.package.subpackage.module.function()" without doing the intermediate imports.
This means NumPy is optimized for programmers (especially novice programmers) using NumPy in long-lived processes where startup cost is a negligible overhead.
Which isn't all use-cases for numeric computing.
15 years ago I supported a CGI-based web app. It was very important to pull out all the stops (delay imports until needed, use zip packages) because it was easier to do that than to re-write everything for another architecture.
The dog does count pretty well after all.
> It's the job of library and app authors to update.
Why? Linus Torvalds doesn't agree with you, for one.
As Hinsen points out,
] Unfortunately, the need for long-term stability is rather specific to scientific users, and not even all of them require it (see e.g. these two tweets by Titus Brown). So while Python 3 is probably a step forward for most Python users, it’s mostly a calamity for computational science.
Some scientific code has been able to run unchanged since the 1970s, through multiple new Fortran language releases.
Now, yes, I know the reasons for the changes to Python. I know the funding and organizational realities.
But why not recognize that for some situations Python 3 is not better?
Hinsen also comments on your proposal:
] The implication is that breaking changes in the infrastructure layers are OK and must be absorbed by the maintainers of layers 3 and 4. In view of what I just said about layer 4, it should be obvious that I don’t agree at all with this point of view. But even concerning layer 3, I find it a bit arrogant. The message to research communities with weaker code development traditions, and thus fewer resources, is that their work doesn’t matter.
> Could Hinsen (and others) not just version their deps?
He addresses that, I think. One of the other commenters gives a more complete reply at https://metarabbit.wordpress.com/2017/11/18/numpy-scipy-back... ending "Freezing the versions solves some problems, but does not solve the whole issue of backwards compatibility.".
> Time to forget all about this and build cool stuff, please please please.
I'll quote Hinsen again "I find it a bit arrogant. The message to research communities with weaker code development traditions, and thus fewer resources, is that their work doesn’t matter."
Your implicit statement is that mmtk (Hinsen's code base) isn't "cool stuff". Why? Simply because it's old, or because you don't know about it or need it? What other cool old stuff will die because it's part of a community without the resources to update?
Instead, accept that that loss is part of the trade-offs, be empathetic to those who suffer, and bear those lessons in mind for future work you do.
Second, I admit to engaging in hyperbole when I said "Python 3 is better in every way"; usually I'm on the other side of these, but I'm just so fed up with people complaining. But you're right, there are still ways Python 3 isn't "better". I'd love to have productive, technical discussions about them, but we can't seem to get beyond the "Python 3 was a super bad idea" stuff, and I'm totally uninterested in that.
But beyond that, you and I are mostly talking about different things. Python 3 isn't NumPy or SciPy. If you're building extensions on top of them, you need to look at their compatibility commitments. If you want them to make more commitments, you have to convince them. This isn't specific to software engineering; this is due diligence for anything you're gonna put years of work into.
Django's page [1] is a great example of this. Python has one too [2]. I don't have any idea about SciPy/NumPy; It looks like SciPy 1.2.0 was an LTS release supported until 1/1/2020, but what do I know.
But importantly, the end result of this "hey, do 100x the work otherwise our science won't be reproducible" stuff will be to force people out of producing free software for scientific computing. And the non-free stuff is expensive, good god. Surely this isn't what you want.
A better tactic here is to work with the developers in establishing more compatibility between releases. You probably aren't gonna get Fortran levels of compatibility--a language and platform that's seen very, very little change over the decades. But then again, the core selling point of scientific Python is that you get to use a modern platform with modern features. Asking for that along with a 50 year compatibility guarantee is a laughably tall order: you can't have it both ways without exponential amounts of work. So just like you're asking other engineers to be empathetic and respect your need for more compatibility with your extensions, you need to be more empathetic and respect their resources. And the best place to do that is probably their contact page [3], not Twitter, HN, or random blogs.
[1]: https://www.djangoproject.com/download/#supported-versions
Perhaps your "fed up"-ness means you overlook conversations which do go beyond that? Or do you put me into that category as well?
> Python 3 isn't NumPy or SciPy. ... this is due diligence for anything you're gonna put years of work into.
Hinsen's essay discussed these issues related to "software layers and the lifecycle of digital scientific knowledge". He put Python in layer 1, and NumPy/Scipy in layer 2.
In his essay he also said "I would like to see the SciPy community define its point of view on these issues openly and clearly. ... It’s OK to say that the community’s priority is developing new features and that this leaves no resources for considering stability. But then please say openly and clearly that SciPy is a community for coding-intensive research and that people who don’t have the resources to adapt to breaking changes should look elsewhere. Say openly and clearly that reproducibility beyond a two-year timescale is not the SciPy community’s business, and that those who have such needs should look elsewhere."
So I'm not convinced that we are talking about different things as you are making points I already referred to, albeit indirectly.
I'm also not sure you understood all of Hinsen's points. I say this because you wrote ""hey, do 100x the work otherwise our science won't be reproducible" stuff"
But Hinsen said "Layer 4 code is the focus of the reproducible research movement" and "the best practices recommended for reproducible research can be summarized as “freeze and publish layer 4 code” -- a solution you mentioned earlier.
It's just that reproducibility isn't the only goal for stability.
Another is to be able to go back to a 15 year old project and keep working on it, without taking the hit of rewriting it to a new, albeit similar, language.
I also have a small amount of umbrage about your comment:
> So just like you're asking other engineers to be empathetic and respect your need for more compatibility with your extensions, you need to be more empathetic and respect their resources.
I earlier wrote "Now, yes, I know the reasons for the changes to Python. I know the funding and organizational realities."
Did you overlook that because of your '"fed up"-ness', or was that not enough for you?
I do put you in that category, because you seem to be focused much more on the negative, rather than being constructive and trying to find solutions to problems.
> I'm also not sure you understood all of Hinsen's points. I say this because you wrote ""hey, do 100x the work otherwise our science won't be reproducible" stuff"
I've read and directly disagreed with his essay. His points are:
- Python 2 going away orphans a lot of software, because there's a lack of resources/willingness to port to Python 3.
- Python 3 didn't provide enough value to the scientific community to justify all the breakage (this is true for almost every community, btw).
- SciPy breaks compatibility roughly every 2-3 years, which is a bad fit for the pace of scientific computing.
- Beyond that, breaking compatibility threatens reproducibility.
- The SciPy community doesn't seem to know or care about compatibility concerns.
- Projects written on top of SciPy libraries ("Layer 3" code) have to keep updating, and they don't always have resources/willingness to do that.
- It would be cool if SciPy laid out a support schedule.
- It isn't cool that SciPy says, "hey use us", and then breaks compat all the time.
- There are some languages/platforms that haven't changed in decades, this isn't an excuse.
Here's what I've said:
- Agree Python 3 didn't provide enough value.
- If you want to build something on SciPy that you expect to last for decades, you should look for a compat guarantee. If you don't, that's on you.
- If you want new features plus decades of compat, that's a ludicrous amount of work.
- If you want to find a way forward, start a dialogue with SciPy devs.
Hinson's examples of Fortran and Java are illuminating. Fortran's a platform that's seen very minimal evolution over its history. That's exactly the reason people want to use SciPy instead of Fortran. Java's a platform with... billions of engineering hours? It's ironic that a guy who doesn't want to spend the resources to update his own software is asserting that someone else can continually deliver a modern scientific computing platform with new features while never breaking compat, they just don't feel like it ("It's all a matter of policy, not technology"). That's wrong, it's a question of resources.
---
My diagnosis here is communication breakdown. Everyone here wants the same thing: use a modern software stack for scientific computing. So again I'll say get on the mailing lists, get on IRC, go to the conferences, and talk to the engineers. Be constructive.
I'll admit Python 3 is still slower at a lot of things. But that feels like saying your new dog is even worse at math than your old one.
The C extension thing isn't Python's fault. It's the job of library and app authors to update. Do we complain that Vulkan has bad SunOS support? This is totally backwards.
Could Hinsen (and others) not just version their deps? It's not like people are erasing Python 2 off the internet. If his main worry is reproducibility, he should be doing that anyway.
---
I don't want to give the impression I like the whole Python 3 thing. I think it was a pretty big mistake and a huge missed opportunity. I'm very sympathetic to people who had to put in a lot of work for basically no good reason--Python 3 didn't really offer anything significantly better than 2 until... 3.5 (3.4 if you think the first pass at async was useful, I personally don't).
But I also find the ballyhooing about it really insufferable. Yeah it was a mistake; Armin Ronacher (as usual) was right. It was also over 11 years ago. Time to forget all about this and build cool stuff, please please please.