Re:Python idioms
by Estragon on Sunday June 12, 2011 @02:16PM
(#2647025)
> [Despite the existence of "enumerate", "xrange(maxint)"] can still
> be useful when you want to include an index along with several
> other lists, however, e.g. zip(list_1, list_2, indices)
I would just use
for idx, (elt1, elt2) in enumerate(zip(list_1, list_2))
The following is not necessarily good advice, and he should take his own earlier advice and validate his claims with some profiling.
> [map is ] much faster, since the loop takes place entirely in
> the C API and never has to bind loop variables to Python
> objects.
> If you find yourself making the same list comprehension
> repeatedly, make utility functions and use map and/or filter...
Note that the following "map" snippet is considerably slower than the corresponding list-comprehension snippet. Function calls are expensive in python. Also, his claim that you save time with a "map" by avoiding the loop variable is obviously bogus: you still have to bind the variables in the signature of the function you're mapping, unless the signature is empty.
met% python -m timeit "[x**3 for x in xrange(10000)]"
1000 loops, best of 3: 1.27 msec per loop
met% python -m timeit "map(lambda x: x**3, xrange(10000))"
1000 loops, best of 3: 1.88 msec per loop
Re:Python idioms
by orlandu63 on Sunday June 12, 2011 @03:33PM
(#2647196)
I'm getting entirely different results with Python 3.2:
% python -m timeit "[ x**3 for x in range(10000) ]"
100 loops, best of 3: 7.85 msec per loop
% python -m timeit "map(lambda x: x**3, range(10000))"
1000000 loops, best of 3: 1.26 usec per loop
By these benchmarks map is 700 times faster than the corresponding list comprehension.
Re:Python idioms
by imurray on Sunday June 12, 2011 @03:48PM
(#2647228)
When times are that short you should question if Python is actually doing anything, or just promising to do it later (remember that range is a generator in python3).
$ python3.1 -m timeit "map(lambda x: x**3, range(10000))"
1000000 loops, best of 3: 1.01 usec per loop
Seems fishy. Adding up the elements forces Python to actually do the cubing on all of them:
$ python3.1 -m timeit "sum(map(lambda x: x**3, range(10000)))"
100 loops, best of 3: 13 msec per loop
$ python3.1 -m timeit "sum([ x**3 for x in range(10000) ])"
100 loops, best of 3: 10.6 msec per loop
map really is slower, at least for me in python3.1.
Re:Python idioms
by imurray on Monday June 13, 2011 @01:29PM
(#2649853)
[ldng: put some whitespace before code on hacker news: it indents it and the asterisks don't disappear.]
I infer that ldng has a 64 bit machine. On my 32 bit machine, Python 3.1 is faster than 2.6 for these examples. On a 64 bit machine I get similar results to ldng's, with Python3 being slower. If I wrap long() around the x in the example, Python2 becomes as slow as Python3.
Note that taking the cube of lots of big integers is not typical for many people: it generates very large integers that have to be in Python2's special long type on a 32 bit machine. On a 64 bit machine they stay as normal ints in Python2, which are much faster. Python3 has a single automagic int type, which seems to internally convert to the arbitrary precision type sooner than it has to on 64 bit machines(?).
Examples more typical of my use would wrap float() around the x, or change the example to add up 3x instead of x^3. These examples are all faster in Python3 for me. Faster still is to use numpy (which is now supported in Python3).
Summary: the people who would be affected by this regression have both a 64 bit machine, and do a lot of exact integer arithmetic on integers that can be represented in 64 bits, but not 32.
Re:Python idioms
by deno on Sunday June 12, 2011 @03:51PM
(#2647235)
map() returns an iterator in Py3k.
> python3.2 -m timeit "( x**3 for x in range(10000))"
100000 loops, best of 3: 2.67 usec per loop
> python3.2 -m timeit "[ x**3 for x in range(10000)]"
100 loops, best of 3: 9.53 msec per loop
> python3.2 -m timeit "list(map(lambda x: x**3, range(10000)))"
100 loops, best of 3: 11.8 msec per loop
> python3.2 -m timeit "list( x**3 for x in range(10000))"
100 loops, best of 3: 10.7 msec per loop
And here's the Python 2.5 naïve/in-memory version recreated in Python 3.2:
> python3.2 -m timeit "list(map(lambda x: x**3, list(range(10000))))"
100 loops, best of 3: 12.1 msec per loop
And on Python 2.7 is still the fastest:
> python -m timeit "map(lambda x: x**3, range(10000))"
100 loops, best of 3: 4.33 msec per loop
> python -m timeit "[x**3 for x in xrange(10000)]"
100 loops, best of 3: 2.32 msec per loop
> python -m timeit "[x**3 for x in range(10000)]"
100 loops, best of 3: 2.66 msec per loop
On PyPy (1.5) it's pretty much the same, which goes to show why this is not “Python idiom”, but rather CPython's implementation detail:
> pypy -m timeit "[x**3 for x in range(10000)]"
100 loops, best of 3: 2.73 msec per loop
> pypy -m timeit "[x**3 for x in xrange(10000)]"
100 loops, best of 3: 2.62 msec per loop
> pypy -m timeit "map(lambda x: x**3, xrange(10000))"
100 loops, best of 3: 2.51 msec per loop
> pypy -m timeit "map(lambda x: x**3, range(10000))"
100 loops, best of 3: 2.87 msec per loop
Did you try itertools.imap instead? Vanilla map gives us a list; imap gives us a generator. (Also, I'd try a generator expression as well to see how it compares to a list comprehension).
Also, note that the article is from January 2007; I'd wager that Python has evolved and had a wealth of optimisations since then, especially regarding "common idioms" such as vanilla string concatenation and list comprehensions.
Re:Python idioms
by Estragon on Sunday June 12, 2011 @04:56PM
(#2647378)
That's not strictly correct. It really comes down to whether you're calling a native python function or a C extension. If it's a native python function call, it's going to be relatively slow.
Re:Python idioms
by Ysx on Sunday June 12, 2011 @01:58PM
(#2646966)
Good stuff, though there's a few outdated idioms:
Lists, or any iterable, can be reverse-sorted with reversed(sorted(my_list)). That'll give you an iterator, though you can call list() on the result if you need it.
"while True" should be used in-place of "while 1". Reads better.
In Python 2, xrange() is preferred over range() when looping - it won't create an in-memory list of integers, and behaves mostly the same as range but for a few edge cases. Python 3 renamed xrange() to range(), and removed the original range() function.
Re:Python idioms
by imurray on Sunday June 12, 2011 @02:14PM
(#2647015)
Always worth checking. Although the relative merits change when doing many joins to build up a single long string.
This recommendation is to go with .join() by default:
In performance sensitive parts of the library, the ''.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations. — http://www.python.org/dev/peps/pep-0008/
despite the fact it might not be better in CPython.
It's not terribly surprising. Things that scale faster tend to have higher setup costs (cf sorting algorithms; insertion sort is the fastest for relatively small n).
Re:Python idioms
by thristian on Sunday June 12, 2011 @11:18PM
(#2647988)
I believe the Python runtime has special-case handling for concatenating string-literals, and for concatenating string whose reference count is exactly 1. String concatenation isn't wholly defanged, though.
Re:Python idioms
by 1amzave on Sunday June 12, 2011 @07:26PM
(#2647635)
> Calling it from the empty string concatenates the pieces with no separator, which is a Python quirk and rather surprising at first.
This is stated twice, but I can't make any sense of it. How is this even remotely surprising? What else would anyone possibly expect joining with the empty string as the separator to do?
Re:Python idioms
by skimbrel on Sunday June 12, 2011 @08:11PM
(#2647703)
Use function factories to create utility functions. Often, especially if you're using map and filter a lot, you need utility functions that convert other functions or methods to taking a single parameter. In particular, you often want to bind some data to the function once, and then apply it repeatedly to different objects. In the above example, we needed a function that multiplied a particular field of an object by 3, but what we really want is a factory that's able to return for any field name and amount a multiplier function in that family:
def multiply_by_field(fieldname, multiplier):
"""Returns function that multiplies field "fieldname" by multiplier."""
def multiplier(x):
return getattr(x, fieldname) * multiplier
return multiplier
triple = multiply_by_field('Count', 3)
quadruple = multiply_by_field('Count', 4)
halve_sum = multiply_by_field('Sum', 0.5)
Other languages (most prominently Haskell, though you can convince most Lisps to do it through the clever use of macros) have built-in support for doing this with whatever function you want, and it's called partial function application. It's a rather useful technique and it saddens me that it's not supported as such in more languages claiming to support functional programming.
Re:Python idioms
by herdrick on Monday June 13, 2011 @05:40PM
(#2650835)
You don't need a macro to do that in a lisp. Typically you just use a built in 'partial' function, which just makes a closure that will call the original function. (It's simple, so you can just make your own if you're using a lisp without it built in.)
Re:Python idioms
by j_baker on Sunday June 12, 2011 @08:19PM
(#2647722)
Use if not x instead of if x == 0 or if x == "" or if x == None or if x == False; likewise, if x instead of if x != 0, if x != None, etc.
Be careful with this one. "if not x" isn't necessarily the same as "if x == None". It's easy to forget that "if not x" will be true for values other than None.
Also, use "if x is None" rather than "if x == None". :-)
I read many years ago that '%s%s' % (a,b) was faster than a + b, the reason given that it was done in C. But after reading this thread and trying myself, it seems to be false also. On Py 2.6:
python -m timeit " '%s%s%s' % ('aaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbbb', 'ccccccccccccccccccccccccccccccc') "
1000000 loops, best of 3: 0.201 usec per loop
python -m timeit " 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + 'bbbbbbbbbbbbbbbbbbbbbbbb' + 'ccccccccccccccccccccccccccccccc' "
10000000 loops, best of 3: 0.136 usec per loop
Perhaps it changed in new versions of Python, but either way, I guess I'll be using the % form less than I used to.
Re:Python idioms
by fijal on Monday June 13, 2011 @08:51AM
(#2648836)
For what is worth, most "performance" idioms are anti-idioms on PyPy. Especially those:
sum = 0
for d in data:
sum += d
product = 1
for d in data:
product *= d
Are much faster than reduce/map equivalents. The zip/dict example at the end is even more confusing. I'm convinced PyPy would be the fastest on the simplest-possible code (the first one, marked as "bad")
http://www.artima.com/weblogs/viewpost.jsp?thread=98196
> for idx, (elt1, elt2) in enumerate(zip(list_1, list_2))
or use itertools.count:
100 loops, best of 3: 7.61 msec per loop
$ python3.2 -m timeit "sum([ x3 for x in range(10000) ])"
100 loops, best of 3: 6.27 msec per loop
$ python2.7 -m timeit "sum(map(lambda x: x3, range(10000)))"
100 loops, best of 3: 2.81 msec per loop
$ python2.7 -m timeit "sum([ x3 for x in range(10000) ])"
100 loops, best of 3: 2.28 msec per loop
I wasn't expecting python3 to be that slower.
I infer that ldng has a 64 bit machine. On my 32 bit machine, Python 3.1 is faster than 2.6 for these examples. On a 64 bit machine I get similar results to ldng's, with Python3 being slower. If I wrap long() around the x in the example, Python2 becomes as slow as Python3.
Note that taking the cube of lots of big integers is not typical for many people: it generates very large integers that have to be in Python2's special long type on a 32 bit machine. On a 64 bit machine they stay as normal ints in Python2, which are much faster. Python3 has a single automagic int type, which seems to internally convert to the arbitrary precision type sooner than it has to on 64 bit machines(?).
Examples more typical of my use would wrap float() around the x, or change the example to add up 3x instead of x^3. These examples are all faster in Python3 for me. Faster still is to use numpy (which is now supported in Python3).
Summary: the people who would be affected by this regression have both a 64 bit machine, and do a lot of exact integer arithmetic on integers that can be represented in 64 bits, but not 32.
Also, note that the article is from January 2007; I'd wager that Python has evolved and had a wealth of optimisations since then, especially regarding "common idioms" such as vanilla string concatenation and list comprehensions.
Lists, or any iterable, can be reverse-sorted with reversed(sorted(my_list)). That'll give you an iterator, though you can call list() on the result if you need it.
"while True" should be used in-place of "while 1". Reads better.
In Python 2, xrange() is preferred over range() when looping - it won't create an in-memory list of integers, and behaves mostly the same as range but for a few edge cases. Python 3 renamed xrange() to range(), and removed the original range() function.
I thought sorted(my_list, reverse=True) would be slightly faster, but it seems not (by a tiny amount). Weird.
> "while True" should be used in-place of "while 1". Reads better.
If you disassemble these statements, you'll see that "while 1" creates fewer instructions:
Edit: This is in Python 2.7.1. d0mine pointed out that the discrepancy is no longer the case in Python 3. Good to know. :)10000000 loops, best of 3: 0.026 usec per loop
~$python -mtimeit "''.join(('a','b','c','d'))"
10000000 loops, best of 3: 0.197 usec per loop
$ python -mtimeit "'aaaaaaaaaaaaaaa' + 'bbbbbbbbbbbbbbb' + 'ccccccccccccccc' + 'ddddddddddddddd'"
1000000 loops, best of 3: 0.224 usec per loop
$ python -mtimeit "''.join(('aaaaaaaaaaaaaaa','bbbbbbbbbbbbbbb','ccccccccccccccc','ddddddddddddddd'))"
10000000 loops, best of 3: 0.201 usec per loop
This recommendation is to go with .join() by default:
In performance sensitive parts of the library, the ''.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations. — http://www.python.org/dev/peps/pep-0008/
despite the fact it might not be better in CPython.
For a fair test, add elements of an array vs. joining them.
This is stated twice, but I can't make any sense of it. How is this even remotely surprising? What else would anyone possibly expect joining with the empty string as the separator to do?
Be careful with this one. "if not x" isn't necessarily the same as "if x == None". It's easy to forget that "if not x" will be true for values other than None.
Also, use "if x is None" rather than "if x == None". :-)