Re:Accelerating Python Libraries with Numba (Part 2)
by gcr on Thursday May 23, 2013 @04:43PM
(#5759422)
One possible reason: Numpy arrays are generally strongly typed, which allows Numpy's custom C implementation of sum to work while only dispatching types once instead of for each element.
[ Parent
| on HN ]
Re:Accelerating Python Libraries with Numba (Part 2)
by onalark on Thursday May 23, 2013 @05:01PM
(#5759517)
Gold star for you :) The NumPy implementation is basically a for loop. The only cost is the very brief inspection of the dtype to dispatch on.
Re:Accelerating Python Libraries with Numba (Part 2)
by kingkilr on Thursday May 23, 2013 @06:20PM
(#5759910)
Why can't normal pythons sum do that?
Re:Accelerating Python Libraries with Numba (Part 2)
by rguillebert on Friday May 24, 2013 @07:59AM
(#5762204)
Because CPython has no JIT :)
Re:Accelerating Python Libraries with Numba (Part 2)
by Ihmahr on Friday May 24, 2013 @07:54AM
(#5762194)
Because python lists can contain different types, unlike numpy arrays
Re:Accelerating Python Libraries with Numba (Part 2)
by gcr on Friday May 24, 2013 @11:41AM
(#5763510)
Exactly. To further elaborate, try the following in python, if you have the memory:
>>> x = list(range(100000000)) + ["break"]
After some time, you get a list with a hundred million integers with a string at the end. >>> len(x)
100000001
Now take the sum: >>> sum(x)
On my machine, I get a traceback, but after about half a second of hesitation. Python's been iterating through each of those elements, testing to see if it's an integer and building a running sum, only to throw an exception because it didn't expect the string at the end. That means that in order to avoid memory corruption (or other terrible fates worse than an exception), Python must double-check each element of the list as it goes; it cannot take shortcuts.
Re:Accelerating Python Libraries with Numba (Part 2)
by kingkilr on Friday May 24, 2013 @04:13PM
(#5764969)
Can't Python notice that a list always has the same types and do something special?
Re:Accelerating Python Libraries with Numba (Part 2)
by Ihmahr on Saturday May 25, 2013 @07:42AM
(#5766963)
That could have been implemented but it is not. I dont think it will be because it is not that usefull and would be a lot of work/code for the small benefits. If you want faster python use pypy, if you want homogenieus lists you probably could use numpy arrays.