Re:Hunting Performance in Python Code: Measuring Memory Cons...
by falcolas on Thursday June 01, 2017 @10:44AM
(#14460662)
Well, the fact that they're running it in PyPy makes all of this that much more challenging to debug - you'd have to look at the runtime-generated bytecode to identify the actual causes.
Re:Hunting Performance in Python Code: Measuring Memory Cons...
by lilbobbytables on Thursday June 01, 2017 @10:47AM
(#14460689)
Localizing append helps, but that's not all of it:
Building a list of 10,000,000 items:
List comprehension: 599ms For loop: 930ms For loop with append localized: 736ms
https://wiki.python.org/moin/PythonSpeed/PerformanceTips#Loo...
I imagine that lookup overhead would happen on every iteration, but the list add only happens when the conditional evaluates to true - so it seems a bit less likely that the append is the overhead, but maybe the cost of allocating memory is so bad (it is a syscall after all) that it compensates for the infrequency.
Generators would help with this too: instead of materializing and appending the list at the end every time (i.e. [2] + [x for x in s if x]), you could do something like itertools.chain([2], (x for x in s if x)). Then you avoid ever allocating a massive list -- provided that the rest of your code avoids that too.