News for hackers. Stuff that matters.
Re:Optimizations which made Python 3.6 faster than Python 3.... by nomel on Monday June 05, 2017 @10:45PM (#14494313)
I think the problem is pep8. For some reason, pep8 says you load your modules at the top of the file rather than when needed. Giant libraries often follow pep8, meaning something like "import pandas" loads hundreds of python files, which gives the absurd startup times.

I've seen some large code bases take 4 seconds to import, > 80% being unused code.

Re:Optimizations which made Python 3.6 faster than Python 3.... by vosper on Tuesday June 06, 2017 @01:31AM (#14494973)
Huh, I never thought about that before. Importing inside a function is very frowned-upon AFAIK, but now I wonder what kind of performance improvements you might get from importing things only when you need them.

I wonder if it would be possible to automatically rewrite a script to do that.

Re:Optimizations which made Python 3.6 faster than Python 3.... by dragonwriter on Tuesday June 06, 2017 @11:18PM (#14503479)
> For some reason, pep8 says you load your modules at the top of the file

Clarity of dependencies is the “some reason”.

Like most style rules, there are times when breaking it is justified by other considerations, but it's not an arbitrary rule.

Re:Optimizations which made Python 3.6 faster than Python 3.... by smitherfield on Monday June 05, 2017 @11:28PM (#14494531)
But couldn't that inefficiency be optimized away in the implementation? (Load imports lazily).
Re:Optimizations which made Python 3.6 faster than Python 3.... by marmaduke on Tuesday June 06, 2017 @01:21AM (#14494927)
It can't be done automatically because imported can execute code. Since that's part of the semantics, you have to be lazy by hand.
Re:Optimizations which made Python 3.6 faster than Python 3.... by deckiedan on Tuesday June 06, 2017 @01:34AM (#14494991)
Only on the first load ever. When modules are compiled to their byte code, they could get a "clean" flag if they don't execute code, which means they get loaded lazily next time.
Re:Optimizations which made Python 3.6 faster than Python 3.... by marmaduke on Tuesday June 06, 2017 @01:53AM (#14495070)
Ok but figuring out what executes code might require executing code: consider non trivial use of meta classes, where an inconspicuous subclass actually invokes e.g. A registration process like a Django model class.
Re:Optimizations which made Python 3.6 faster than Python 3.... by smitherfield on Tuesday June 06, 2017 @03:19PM (#14500302)
Sure, but, just spitballing here, could a substantial fraction of imported code benefit even with a very conservative heuristic?

In any case, given your points, it seems like a future Python version ought to introduce a new version of the import statement with lazy semantics (which, besides eliminating dead code, is also IMO the more correct/explicit behavior when importing symbols).

Re:Optimizations which made Python 3.6 faster than Python 3.... by marmaduke on Friday June 09, 2017 @02:55PM (#14523540)
It'd be pretty easy to do the experiment, since you can override the import logic and do lazy imports, eg.

    class LazyModule:
        def __init__(self, name):
            self.name = name
        def __getattr__(self, key):
            mod = {}
            exec('import %s as mod', mod)
            self.mod = mod['mod']
            return getattr(self.mod, key)

    class LazyImporter:
        def get_module(self, name):
            return LazyModule(name)
Re:Optimizations which made Python 3.6 faster than Python 3.... by deckiedan on Tuesday June 06, 2017 @10:44PM (#14503314)
That's kind of what I was hoping. I suspect a vast proportion of normal Python code doesn't use the meta programming and run on import and so on.
Re:Optimizations which made Python 3.6 faster than Python 3.... by joshuamorton on Wednesday June 07, 2017 @02:47AM (#14504183)
If a module imports anything, it cannot be assumed clean, since any of its imports may execute code, and they will not be available at compile time.