- Python compiles a single module to bytecode at a time (.py to .pyc)
- It interleaves execution and compilation -- imports are dynamic, you can put arbitrary computation between them, and conditionally import, etc.
If you had structs, then you would need to resolve names to offsets at compile time -- e.g p.x is offset 0, p.y is offset 4. That's a form of type information. Types can be defined in one module and used in another, and Python really has no infrastructure for that.
Nor do any other dynamic languages like Ruby, Perl, JavaScript, Lua, or PHP, as far as I can tell. They are all based on hash tables because deferring name lookup until runtime means that I can start executing "now" rather than looking all over the program for types. It probably helps a bit with the REPL too.
The need for type information in a translation unit is also what gives you C's header file mess, so it's not a trivial problem.
The presence of __slots__ does several things. First, it restricts the valid set of attribute names on an object to exactly those names listed. Second, since the attributes are now fixed, it is no longer necessary to store attributes in an instance dictionary, so the __dict__ attribute is removed. Instead, the attributes can be stored in predetermined locations within an array. Thus, every slot attribute is actually a descriptor object that knows how to set/get each attribute using an array index. Underneath the covers, the implementation of this feature is done entirely in C and is highly efficient.
http://python-history.blogspot.be/2010/06/inside-story-on-ne...
https://stackoverflow.com/questions/14118564/how-does-slots-...
The goal of using __slots__ is to save memory; instead of using a .__dict__ mapping on the instance, the class has descriptors objects for each and every attribute
So python still has to look at the class for each attribute access on an instance of Foo (to find the descriptor). Any unknown attribute (say, Foo.ham) will still result in Python looking through the class MRO to search for that attribute, and that includes dictionary searches.
If I have time I'll write an example program to test this. Since I've been curious for awhile.
I've actually forked the Python interpreter to make my highly compatible bash shell faster and smaller, while not completely rewriting it [1].
-----
Even if __slots__ did what you think it does, it doesn't completely solve the problem.
A lookup like "return os.path.exists(foo)" can't use slots, because it's a module and not a class.
In principle, local variable lookups are also slower in Python than say Dalvik (a JVM without a JIT). LOAD_FAST still involves a dict lookup (as opposed to the even slower LOAD_ATTR). It just happens at the beginning of the function once, not on say every iteration of a loop inside the function.
In contrast, a Java compiler will resolve the attribute access at compile time.
PEP 509 might help with avoiding repeated dictionary lookups in this case, but it's very new: https://www.python.org/dev/peps/pep-0509/
Also, if slots did what you think it does, it would be 10x faster than normal attribute lookup, but it's not (according to the benchmark above, and at least one other one I saw.)
[1] http://www.oilshell.org/blog/2017/04/25.html
EDIT: My last two points might seem like a tangent, since they're not really related to structs. But my overall point is that Python doesn't do very much at compile time, so some of that work has to be done at runtime. slots could easily be faster than it is; it's mainly a memory optimization.
Yes, it has to lookup, but the lookup is cached (see typeobject.c).
Also I did a speed test on my machine. My results are the same as the ones I've found through Google: __slots__ is not much faster. It's maybe 20% faster, whereas if it was really a dict lookup vs. index lookup I would expect it to be an order magnitude faster, or at least 5 times faster.
To be clear, I'm claiming that every time you do "p.x" with a class with __slots__, at least one dictionary lookup occurs. It's not like p.x in C or Java.
I could be wrong, but I've never heard the claim you're making, and I don't see any evidence it's true.
I counted the number of times lookdict_string in dictobject.c is called -- call that number D. And then I made N attribute accesses like 'p.x'.
Without slots, if N goes up, then D goes up. With slots, as N goes up, D stays constant.
Thanks for the information -- I'll look into this more.
(However, as mentioned, slots are not more than 20-30% faster on my machine, in this limited benchmark. I feel like this is a double mystery. Or maybe it's just that interpreter loop overhead or some other overhead is the bottleneck, and not dict access.)
So you avoid the dictionary lookup, but you're still following extra pointers.
It is a matter of caring about it, and having JIT/AOT support as part of the standard tooling.
C header file mess is caused by its designers ignoring the work outside AT&T and not bothering to implement a module system.
To be fair to them, they wanted to create a language whose compiler did fit into the limited hardware they had.
It's not their fault that UNIX had so much success and the world largely went there ignoring most of the progress made in programming tools and languages during the 70s.
An OS and systems programming language available almost for free, including source code, versus what a commercial mainframe, its OS and SDK would cost without source code included, sure recipe for success.
This is not correct.
Common Lisp easily supports structs, and is a dynamic language. No need to specify types for structs, although you can optionally do it in CL if you want.
Here's a textbook example (straight from the CLTL2 book...) Here a struct is defined (only one line needed), and then the struct is used.
;; structure definition
(defstruct door knob-color width material)
;; Automatically creates the following function:
;; make-door
;; And accesors (can be used for getting and setting)
;; door-width
;; door-knob-color
;; door-width
;; door-material
;; ----------------------
;; 'instancing' the structure into "my-door"
(setq my-door
(make-door :knob-color 'red :width 5.0))
;; accessing
(door-width my-door) => 5.0
;; setting
(setf (door-width my-door) 43.7)
;; accessing
(door-width my-door) => 43.7
(door-knob-color my-door) => red(Although I honestly doubt that idiomatic Lisp does this. Idiomatic Lisp might be even slower than idiomatic Python, because I've seen people use alists as structs rather than dictionaries.)
Either way, your example doesn't really address what I'm talking about. Python has a struct module too, and it has namedtuple, etc. My claim is that Python doesn't have C-style structs, where attribute names are resolved to indexes AT COMPILE TIME, because it breaks the model of the bytecode compiler and dynamic module system.
Lisp is flexible about compile time vs. runtime, so I won't be surprised if it can do that. But again there's no evidence in your example that it does that.
Any good Common Lisp implementation will use fixed indexes. The Common Lisp compiler can inline the accessor and will be using fixed indexes.
What Lisp compiles usually not do is inline the data in the struct. Thus the struct slots will contain a pointer to the slot value. Exceptions: small integers, characters, ... Those will fit into the slot and will be stored directly in the struct.
That's the point of structures in Common Lisp.
Below is an example in Clozure Common Lisp on a ARM processor. As you can see, the compiler can inline the accessor and uses fixed indexes to access the slots...
CL-USER> (defstruct person name age)
PERSON
CL-USER> (defun test (person)
(declare (person person)
(inline person-name person-age)
(optimize (speed 3) (safety 0) (debug 0)))
(person-name person))
TEST
CL-USER> (disassemble #'test)
(mov imm0 (:$ 19))
(stmdb (:! sp) (imm0 vsp fn lr))
(mov fn temp2)
(vpush1 arg_z) ;[12]
(ldr arg_z (:@ arg_z (:$ 2))) ; name at pos 2
(ldmia (:! sp) (imm0 vsp fn pc))
NIL
CL-USER> (defun test (person)
(declare (person person)
(inline person-name person-age)
(optimize (speed 3) (safety 0) (debug 0)))
(person-age person))
TEST
CL-USER> (disassemble #'test)
(mov imm0 (:$ 19))
(stmdb (:! sp) (imm0 vsp fn lr))
(mov fn temp2)
(vpush1 arg_z) ;[12]
(ldr arg_z (:@ arg_z (:$ 6))) ; age at pos 6
(ldmia (:! sp) (imm0 vsp fn pc))
NILThe main issues are:
1. The C-struct interoperability is cumbersome, since really it's just byte objects with some helper functions to pack and unpack them according to a given definition (so you need both the bytes and the correct format definition in order to turn it back into native Python objects), and
2. Most people already are using one of a few different struct-like things depending on their exact use case and preference, and getting everyone to agree on a One True Struct Object to build in to Python or its standard library is probably never going to happen. Just look at namedtuples -- they were an attempt to take the most struct-like thing in Python (the tuple) and add extra optional struct-like behavior to appease people who were frustrated with the lack of structs, but of course people rant about them for having a weird internal implementation and for having behavior that makes them backwards-compatible with plain tuples.
It has the advantage over the struct module that the elements can be accessed and modified in place.
Tuples don't name values, but that's actually important for performance - Python is an old-fashioned dynamic language that still uses hashtable access for everything, but tuple indexing is a simple array offset. Considering Python's age, reducing that endlessly-nested series of hashtable hits was important back in the day.
Python36 gradual typing also helps here though, and am definitely interested to see if that changes the game a bit
Not true; C++ classes have zero performance overhead over C/C++ structs (because they're just another word for the same thing).
Of course Python classes have a lot of costly features that C++ classes don't have, but the notion of a "class" (with methods, constructors etc) is not one of them.
In C++ ease of use is given up for the sake of performance. In python it is the other way.
class A {
int a; // private
}
struct A {
int a; // public
}