News for hackers. Stuff that matters.
Re:Greplin (YC W10) open sources 10-15x faster protocol buff... by ot on Thursday January 27, 2011 @05:08AM (#2147307)
> It would be interesting to know why a portable, cross-language serialization format beats out the language-specific one.

Because protobuf parses messages with a fixed schema in a very structured format. Pickle, OTOH, is an interpreted bytecode microlanguage used to describe arbitrary python objects (for instance, pickle can call python functions: http://nadiana.com/python-pickle-insecure)

Also, pickle supports references (so if an object is referenced two times in the same serialization stream, it is serialized only once) and this has a cost at serialization time (need to keep the set of seen references)

Re:Greplin (YC W10) open sources 10-15x faster protocol buff... by sigil on Thursday January 27, 2011 @03:26PM (#2149308)
Perl's Storable can also serialize code references which get eval'ed during deserialization, and can also serialize multiple references once, though you have to be more explicit about that. And Storable is still 2x-3x faster than the already quite fast JSON::XS.

It would seem the bytecode interpreter architecture in Pickle is the limiting factor. If anybody has some good profiling data on Pickle though, I'd love to see it.