News for hackers. Stuff that matters.
Re:Why Python Runs Slow, Part 1: Data Structures by kashif on Thursday February 13, 2014 @07:24AM (#7230913)
The pythonic way of creating a Point like data structure is not

  point = {'x': 0, 'y': 0}
but instead, this

  Point = namedtuple('Point', ['x', 'y'], verbose=True)
The resulting class, I believe to be more performant than using a dictionary - but I haven't actually measured this.
Re:Why Python Runs Slow, Part 1: Data Structures by mjschultz on Thursday February 13, 2014 @07:48AM (#7230989)
I'm probably doing it wrong but I measured exactly this yesterday and the dict version was faster:

    $ python -m timeit --setup "from collections import namedtuple; Point = namedtuple('Point', ['x', 'y']); p = Point(x=0, y=0)" "p.x + p.y"
    1000000 loops, best of 3: 0.284 usec per loop
vs.

    $ python -m timeit --setup "p = {'x': 0, 'y': 0}" "p['x'] + p['y']"
    10000000 loops, best of 3: 0.0737 usec per loop
Maybe the use isn't right because I agree with your belief that namedtuple is suppose to be more performant.
Re:Why Python Runs Slow, Part 1: Data Structures by pdonis on Thursday February 13, 2014 @09:35AM (#7231558)
You should be creating a dict vs. a namedtuple instance once, then accessing it many, many times. What you're doing is creating a lot of dict vs. namedtuple instances, then accessing each one only once. That's mostly testing the instance creation time, not the field access time.
Re:Why Python Runs Slow, Part 1: Data Structures by mjschultz on Thursday February 13, 2014 @10:40AM (#7231876)
That code should only create the dict/namedtuple instance once, then access it many, many times.

The creation occurs in the --setup portion. The field accesses occur in the actual looping portion of the timeit code.

Re:Why Python Runs Slow, Part 1: Data Structures by pdonis on Thursday February 13, 2014 @08:53PM (#7236336)
Ah, ok, I see, for some reason my browser wasn't showing the namedtuple code snippet right. Now I need to go look at the namedtuple code to see why accessing the fields by name is slower than accessing them by index.
Re:Why Python Runs Slow, Part 1: Data Structures by the_mitsuhiko on Thursday February 13, 2014 @08:11AM (#7231094)
Yep. Because the dictionary literal gets created via opcodes in the interpreter loop whereas the named tuple pays the price of calling a function and setting up an interpreter frame.
Re:Why Python Runs Slow, Part 1: Data Structures by mjschultz on Thursday February 13, 2014 @10:39AM (#7231864)
The --setup code (creating the namedtuple and dict) is only executed once.

The timed portion is simply the field accesses.

Re:Why Python Runs Slow, Part 1: Data Structures by tterrace on Thursday February 13, 2014 @08:17AM (#7231129)
Here's a neat (and totally unpythonic) way to bring that down a bit:

    python -m timeit --setup "from collections import namedtuple; Point = namedtuple('Point', ['x', 'y']); p = Point(x=0, y=0); get_y = Point.y.fget; get_x = Point.x.fget" "get_x(p) + get_y(p)"
Re:Why Python Runs Slow, Part 1: Data Structures by mapleoin on Thursday February 13, 2014 @07:49AM (#7231000)
I think you're timing the import in there, too. Plus, creating an object of a class rather than using a literal will always be slower.
Re:Why Python Runs Slow, Part 1: Data Structures by mjschultz on Thursday February 13, 2014 @07:52AM (#7231005)
Does the --setup statement really get timed?

My reading of the docs[1] has always led me to believe that it doesn't. For example, "It is possible to provide a setup statement that is executed only once at the beginning"

[1]: http://docs.python.org/2/library/timeit.html

Re:Why Python Runs Slow, Part 1: Data Structures by vibragiel on Thursday February 13, 2014 @08:20AM (#7231148)
It doesn't get timed. You can quickly test this by adding an expensive operation to your setup and checking that it doesn't affect the resulting time.
Re:Why Python Runs Slow, Part 1: Data Structures by mapleoin on Thursday February 13, 2014 @10:42AM (#7231890)
Aaah, ok. I totally didn't read the --setup arg.
Re:Why Python Runs Slow, Part 1: Data Structures by NigelTufnel on Thursday February 13, 2014 @07:42AM (#7230961)
I did a stupid microbenchmark and it turned out that simple class with __slots__ was the fastest method.
Re:Why Python Runs Slow, Part 1: Data Structures by copx on Thursday February 13, 2014 @09:42AM (#7231599)
While we are at stupid microbenchmarks. The following loop in C (line being a struct, length is int):

  for (int i = 1; i <= 400000000; ++i) {
    ++line.length;
  }
Compiles to the following ASM:

  @8:
  inc eax
  dec edx
  jne @8
..as expected. The compiler realized that there is no need to actually copy the intermediate values to line.length and does everything in registers.

Now here is the same loop in Lua (everything dynamically typed, line.length must be hashed (in theory) just like in Python):

  for i = 1, 400000000 do
    line.length = line.length + 1
  end
LuaJIT 2.1 generates the following ASM:

  ->LOOP:
  addsd xmm7, xmm0
  movsd [rax], xmm7
  add edi, +0x01
  cmp edi, 0x17d78400
  jle 0x7fee13effe0  ->LOOP
The C program executes in ~0.3s, the Lua one in ~0.5s .. and those 0.5s include LuaJIT's startup, profiling, and JIT compilation time. So for a longer running program the difference would be even smaller.

Tl;dr: modern JIT compilers are amazing and can optimize away the things mentioned in the article.

Re:Why Python Runs Slow, Part 1: Data Structures by StillBored on Thursday February 13, 2014 @04:33PM (#7234536)
I think you forgot to turn on your optimizer. GCC will optimize that loop into something along the lines of

  line.length+=400000000;
That is because it can predict the loop result during compile time.

For example, this is the entire program with your loop and a line to print the result (gcc 4.3.4/linux -O3),

  mov    $0x17d78401,%esi
  mov    $0x400654,%edi
  xor    %eax,%eax
  jmpq   0x400430 <printf@plt>
In fact without the printf all you get is "reqz ret" (not counting .init overhead in the binary). That is because the compiler detects that line.length is not used and fails to even set it.

  #include <stdio.h>

  int main(int argc,char *argv[])
  {
	struct lines
	{
		char *somedata;
		int length;
	} line;
	int i;

        line.length=0;

        for (i=0;i<=400000000;++i)
	{
		++line.length;
	}

        printf("Line len %d\n",line.length);
  }
Re:Why Python Runs Slow, Part 1: Data Structures by copx on Friday February 14, 2014 @02:03AM (#7237351)
>I think you forgot to turn on your optimizer.

I did not. I used Pelles C to compile it (with the optimizer turned on). I am not surprised that GCC managed to eliminate the pointless loop entirely in this situation. In fact I was happy that both Pelles C and LuaJIT did not realize that the whole loop was pointless and thus I did not have to come up with a more complex example.

The primary point of this was to show that JIT compilers can optimize away hash table access and dynamic typing in hot loops, not a code generation competition between LuaJIT and GCC.

I am a pretty big LuaJIT fanboy and not even I would claim that Lua compiled with LuaJIT can compete against C compiled by current versions of GCC with all optimizations turned on, at least not in most real-world cases. However, it does get amazingly close, most of the time within an order of magnitude, which means that I personally do not need to use C anymore.

Re:Why Python Runs Slow, Part 1: Data Structures by michaelmior on Thursday February 13, 2014 @09:01AM (#7231369)
Curious if you included ctypes.Structure in your benchmark.
Re:Why Python Runs Slow, Part 1: Data Structures by NigelTufnel on Thursday February 13, 2014 @09:24AM (#7231502)
I just did. And either a) I'm doing something terribly wrong or b) ctypes.Structure is slower than Python class with __slots__.

I would bet on a) - I've never worked with ctypes before.

EDIT: I bet on a) and I was wrong: ctypes offers no speed advantage.

Re:Why Python Runs Slow, Part 1: Data Structures by adnam on Thursday February 13, 2014 @07:42AM (#7230960)
Would't it be more like this?

    points = [(0, 0), (3, 4), ...]
    sum_x = sum([x for x,_ in points])
    sum_y = sum([y for _,y in points])
Re:Why Python Runs Slow, Part 1: Data Structures by tostitos1979 on Thursday February 13, 2014 @07:43AM (#7230965)
I'm curious where one would go to learn specifically about performance issues in Python - patterns to avoid, and those to stick to. Any recommendations?
Re:Why Python Runs Slow, Part 1: Data Structures by mouhtasi on Thursday February 13, 2014 @09:53AM (#7231645)
Re:Why Python Runs Slow, Part 1: Data Structures by ramchip on Thursday February 13, 2014 @07:35AM (#7230945)
It's mentioned in the article, albeit at the end.
Re:Why Python Runs Slow, Part 1: Data Structures by freyrs3 on Thursday February 13, 2014 @05:07PM (#7234794)
> The resulting class, I believe to be more performant than using a dictionary - but I haven't actually measured this.

If you didn't measure it then how did you come to believe this?

Re:Why Python Runs Slow, Part 1: Data Structures by a13xnet on Thursday February 13, 2014 @07:28AM (#7230926)
Hehe, came here to say exactly that.