Re:Serializing Data - JSON vs. Protocol Buffers
by lacker on Monday August 17, 2009 @08:35PM
(#769460)
The internal standard for when speed of protocol buffer operations matters is to use the C++ protocol buffer code but use swig to wrap it for python. Unfortunately, this is really easy internally due to protocol buffer integration with our build system, but not easy externally since most people don't have a build phase for their python projects.
Re:Serializing Data - JSON vs. Protocol Buffers
by jasonwatkinspdx on Monday August 17, 2009 @09:28PM
(#769537)
Large systems tend to push you towards tradeoffs other people wouldn't make. Scalability over efficiency. Bisection bandwidth is the most scarce resource in a large cluster so I imagine protocol buffers have had much more attention paid to saving bytes than to encoding/decoding speed. Latency critical stuff probably doesn't use python anyhow.
Re:Serializing Data - JSON vs. Protocol Buffers
by haberman on Tuesday August 18, 2009 @06:26PM
(#771586)
Protobufs are actually pretty fast to encode and decode (in the neighborhood of 200-300MB/s on my core2 desktop, when using the C++ bindings).
It's just the Python implementation that is slow. I'm working on a Python implementation that will be much faster. It's really unfortunately that Protocol Buffers are getting a bad rap due to the current Python implementation.
http://groups.google.com/group/protobuf/browse_thread/thread...
Recently I tried out keyczar, Google's crypto toolkit and found that their Python implementation was too slow by a factor of 100X because they are using a slow random string generator:
http://groups.google.com/group/keyczar-discuss/browse_thread...
It's hard to see how both of these problems could have passed internal benchmarking and made it into live code. Maybe Google isn't using its own Python implementations internally.