News for hackers. Stuff that matters.
Re:Pyrobuf: A Cython Alternative to Google's Python Protobuf... by haberman on Saturday December 19, 2015 @11:53AM (#10764048)
Ah, that's very interesting -- your desire for more "pure" Python comes from a desire for speed. (Some people who wish the generated code was more concrete want this so the generated code is more readable).

I am curious what led you to the conclusion that descriptors and metaclasses were making Python protobuf JIT-unfriendly. I ask because most of the "meta" stuff happens at startup, when you first import the module. It uses a metaclass to generate a bunch of Python methods, but after that they are just regular Python methods, and should be as easy for PyPy to optimize as anything else. There is a bit of reflection happening in some code-paths (like the __init__ method does loop over a list of fields to decide how to initialize them), but the metaclass at least is pretty much gone after import.

For this reason, I suspect that even if we ditched all the metaclass stuff, you'd see PyPy performance pretty close to what you're seeing now. The only thing I could see potentially making a more significant difference is if there were generated parsing code that switches on field number, instead of looking up fields by number in a dictionary. But given that Python doesn't have an actual switch statement, this might not be an improvement at all.

Re:Pyrobuf: A Cython Alternative to Google's Python Protobuf... by fijal on Sunday December 20, 2015 @09:46AM (#10767147)
Hi

The python code that's generated is ok, but the problem is that everything is very dynamic. If you want this to work fast on PyPy, it should really generate a bunch of getters and setters that use attribute access and not go through generic __getattr__ that does some dictionary lookups. Additionally a lot of code is written in a way that creates a lot of temporary lists iterating over all fields and double checking stuff - this is bound to be way slower than a simple C/Java stuff that does the very simplest "check type - attribute access" sort of stuff.