News for hackers. Stuff that matters.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by ma2rten on Sunday April 25, 2021 @01:25PM (#26934514)
Protobufs are very important for Google. A significant percentage of all compute cycles is used on parsing protobufs. I am surprised that the parsing is not doing using handwritten assembly if it's possible to improve performance so much.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by CoolGuySteve on Sunday April 25, 2021 @02:57PM (#26935485)
Protobuf's abysmal performance, questionable integration into the C++ type system, append-only expandability, and annoying naming conventions and default values are why I usually try and steer away from it.

As a lingua franca between interpreted languages it's about par for the course but you'd think the fast language should be the fast path (ie: zero parsing/marshalling overhead in Rust/C/C++, no allocations) as you're usually not writing in these languages for fun but because you need the thing to be fast.

It's also the kind of choice that comes back to bite you years into a project if you started with something like Python and then need to rewrite a component in a systems language to make it faster. Now you not only have to rewrite your component but change the serialization format too.

Unfortunately Protobuf gets a ton of mindshare because nobody ever got fired for using a Google library. IMO it's just not that good and you're inheriting a good chunk of Google's technical debt when adopting it.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by haberman on Sunday April 25, 2021 @03:18PM (#26935671)
Zero parse wire formats definitely have benefits, but they also have downsides such as significantly larger payloads, more constrained APIs, and typically more constraints on how the schema can evolve. They also have a wire size proportional to the size of the schema (declared fields) rather than proportional to the size of the data (present fields), which makes them unsuitable for some of the cases where protobuf is used.

With the techniques described in this article, protobuf parsing speed is reasonably competitive, though if your yardstick is zero-parse, it will never match up.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by CoolGuySteve on Sunday April 25, 2021 @03:32PM (#26935817)
Situations where wire/disk bandwidth are constrained are usually better served by compressing the entire stream rather than trying to integrate some run encoding into the message format itself.

You only need to pay for decompression once to load the message into ram rather than being forced to either make a copy or pay for decoding all throughout the program whenever fields are accessed. And if the link is bandwidth constrained then the added latency of decompression is probably negligible.

The separation of concerns between compression format and encoding also allows specifically tuned compression algorithms to be used, for example like when switching zstd's many compression levels. Separating the compression from encoding also lets you compress/decompress on another processor core for higher throughput.

Meanwhile you can also do a one shot decompression or skip compression of a stream for replay/analysis; when talking over a low latency high bandwidth link/IPC; or when serializing to/from an already compressed filesystem like btrfs+zstd/lzo.

It's just more flexible this way with negligible drawbacks.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by nly on Sunday April 25, 2021 @08:09PM (#26937993)
Recently I've been looking at CapnProto which is a fixed offset/size field encoding that allows for zero copy/allocation decoding, and arena allocation during message construction.

One nice design choice it has is to make default values zero on the wire by xor'ing all integral fields with the field default value.

This composes well with another nice feature it has, which is an optional run-length style packed encooding that compresses these zero bytes down. Overall, not quite msgpack efficiency but still very good.

One even more awesome feature is you can unpack the packed encoding without access to the original schema.

Overall I think it's a well designed and balanced feature set.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by Arelius on Sunday April 25, 2021 @09:58PM (#26938503)
I’ve been using CapnProto, and while I like it, it certainly has a small community, and support can suffer due to that. I haven’t tried it, but have heard good thing about flatbuffers, and would def give that a second look if I were to make the decision again.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by nly on Monday April 26, 2021 @03:35AM (#26940210)
For me the clinchers over flatbuffers are:

- No need to build messages 'inside out' / leaves first to get zero copy behaviour. This is huge.

- Fully functional RPC implementation with really interesting latency hiding behaviour via asynchronous promises. It's got most of what I want from gRPC without the horrid googlisms and dependencies.

- CapnProto has a canonical encoding, which makes it suitable for hashing and comparison.

- It has a packed encoding, whereas with Flatbuffers you're going to have to compress with a more hardcore codec like zlib.

- CapnProto supports runtime reflection via a dynamic API. This is useful in cases where you really need it, like when creating generic bindings for dynamically typed languages or use cases. Like protobufs it has a "lite" mode to opt out of this.

- CapnProto JSON support in the library is better, probably due to the above.

Major cons:

- Ugly schema format.

- Not header only (I've not tested whethwr or not there is a positive here with respect to build times)

- The "kj" support library is a bit of a pain in the ass, refusing to use common C++ vocabulary types like string_view or support common STL data structures like vectors or strings. Writing a message in to a std::vector for instance will require a specialization of an interface.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by Cloudef on Sunday April 25, 2021 @11:50PM (#26939086)
Few years back, i actually compared the three serializations, for the data i used ironically raw struct came on top for every benchmark https://cloudef.pw/protobug.png
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by hedora on Monday April 26, 2021 @11:56AM (#26943945)
This matches my experience as well. You should probably add code size to the graphs. Raw struct probably wins there too.

The only thing I’ve seen that’s competitive to raw struct is writing a C++ template visitor style pattern to walk the struct fields using inline methods. (This can be achieved in some other compiled languages, of course.)

It lets you compress things as you walk the struct, and breaks the dependency on struct layout (which can matter for cross-platform stuff, and for versioned protocols). It’s not quite as fast as a memcpy off the network buffer, but it can be branch and virtual dispatch free (not counting branches in the compression logic).

Also, it can validate as it copies the data into the struct, so now you can auto-generate application-specific validation logic. This is a big deal, since validating the output of a deserializer is almost as hard as hand deserialization!

I really liked the article however. Forcing tail optimization in the c++ template serializer sounds like it will substantially improve the (worst case) generated assembly.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by mypalmike on Monday April 26, 2021 @01:58AM (#26939663)
Why is that ironic? Raw struct is the no-op of serialization. Hard to beat zero CPU cycles.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by Cloudef on Monday April 26, 2021 @05:38AM (#26940802)
Performance wise yes, but it also absolutely beat all of the solutions in space used, and no there was no compression involved.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by lmeyerov on Sunday April 25, 2021 @08:13PM (#26938024)
We jumped from protobuf -> arrow in the very beginning of arrow (e.g., wrote on the main lang impls), and haven't looked back :)

if you're figuring out serialization from scratch nowadays, for most apps, I'd def start by evaluating arrow. A lot of the benefits of protobuf, and then some

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by cookguyruffles on Sunday April 25, 2021 @09:00PM (#26938255)
I've played with Arrow a bunch of times and have yet to figure out what it's intended for precisely. (Not joking)
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by VHRanger on Sunday April 25, 2021 @09:44PM (#26938445)
Interchange format for tabular data.

Think pandas in python, but language agnostic.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by lmeyerov on Sunday April 25, 2021 @10:25PM (#26938636)
Yep; fast+safe+well-tooled common IO format for tabular data, especially columnar, for going across runtimes/languages/processes/servers with zero copy. It supports a variety of common types, including nested (ex: lists, records, ...), modes like streaming, and schemas that can be inferred+checked, so it covers a lot of classic protobuf messaging use cases.

When messages look more like data, even better, as columnar representations are often better for compression & compute. So it's pretty common now for data frameworks, and increasingly, DB connectors. Ex: Our client https://github.com/graphistry/pygraphistry takes a bunch of formats, but we convert to Arrow before sending to our server, we zero-copy it between server-side GPU processes, and stream it to WebGL buffers in the client, all without touching it (beyond our own data cleaning).

The main reason I'd stick to protobuf is legacy, especially awkward data schemas / highly sparse data, and I'm guessing, Google probably has internal protobuf tooling that's faster+easier+better tooled than the OSS stuff.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by infinite8s on Tuesday April 27, 2021 @03:14PM (#26960165)
Are you leveraging arrow-flight to stream between server-side to client side? Or some custom protocol embedding the arrow data as a binary blob?
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by lmeyerov on Tuesday April 27, 2021 @03:32PM (#26960395)
I'm probably way out of date, but arrow-flight seems primarily innovative for scaling reads across multiple machines, such as in dask envs

For server->client, arrow's record batch representation is useful for streaming binary blobs without having to touch them, but for the surrounding protocol, we care way more about http/browser APIs and how the protocol works, vs the envs + problems flight was built for. So we built some tuned layers for pushing arrow record batches through GPU -> CPU -> network -> browser. The NodeJS ecosystem is surprisingly good for streaming buffers, so it's a few libs from there, careful protocol use to ensure zero-copy on the browser side, and tricks around compression. Flight, at least at the time, was more in the way than a help for all that, and all arrow sponsor money goes to ursa labs, so we never ended up having the resources to OSS that part, just the core Arrow JS libs. Our commercial side is starting to launch certain cloud layers that may make it more aligned to revisit & OSS those pieces, but for now, still low on the priority list for what our users care about. Our protocol here is about to get upgraded again, but not fundamentally wrt the network/client parts.

Interestingly, for the server case... despite our using dask etc. increasingly on the server, we don't explicitly use arrow-flight, and I don't believe our next moves here will either. I recorded a talk for Nvidia GTC about our early experiments on the push to doing 90+ GB/s per server, and while parquet/arrow is good here, flight still seems more of a distraction: https://pavilion.io/nvidia (with more resources, I think it can be aligned)

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by infinite8s on Tuesday April 27, 2021 @03:58PM (#26960726)
OK thanks for the insight. I havenv't been tracking arrow-flight recently, but it does seem more geared to an RPC use case than one way streaming.

I've been playing with pushing arrow batches using (db -> server-side turbodbc -> network -> browser -> webworker using transferable support), leveraging a custom msgpack protocol over websockets, since most databases don't support connectivity from a browser. Not dealing with big data by any means (max a million records or so), and so far it's been fairly performant.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by lmeyerov on Tuesday April 27, 2021 @04:16PM (#26960949)
yep that's pretty close (and cool on turbodbc, we normally don't get control of that part!)

for saturating network, afaict, it gets more into tuning around browser support for parallel streams, native compression, etc. we've actually been backing off browser opts b/c of unreliability across users, and thus focus more on what we can do on the server..

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by Cloudef on Sunday April 25, 2021 @11:44PM (#26939056)
The protobuf itself as format isnt that bad, just the default implementations are bad. Slow compile times, code bloat and clunky apis / conventions. Nanopb is much better implementation and allows you to control code generation better too. Protobuf makes sense for large data, but for small data fixed length serialization with compression applied on top probably would be better.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by gorset on Sunday April 25, 2021 @03:26PM (#26935744)
It's obviously possible to do protobuf with zero parsing/marshalling if you stick to fixed length messages and 4/8 byte fields. Not saying that's a good idea, since there are simpler binary encodings out there when you need that type of performance.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by ori_b on Sunday April 25, 2021 @09:07PM (#26938288)
This is incompatible with protobuf. Protobuf has variable length encodings for all its integers, including field tags.

https://developers.google.com/protocol-buffers/docs/encoding

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by gorset on Monday April 26, 2021 @02:29AM (#26939842)
Actually you have both 32 and 64 bit wire types:

    - wire_type=1 64 bit: fixed64, sfixed64, double
    - wire_type=5 32 bit: fixed32, sfixed32, float
Consider a valid protobuf message with such a field. If you can locate the field value bytes, you can write a new value to the same location without breaking the message. It's obviously possible to the same with the varint type too, as long as you don't change the number of bytes - not so practical, but useful for enum field which has a limited set of useful values (usually less than 128).

Pregenerating protobuf messages you want to send and then modifying the bytes in-place before sending is going to give you a nice performance boost over "normal" protobuf serialization. It can be useful if you need to be protobuf compatible, but it's obviously better to use something like SBE - https://github.com/real-logic/simple-binary-encoding

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by fnord123 on Sunday April 25, 2021 @04:21PM (#26936296)
FWIW, the python protobuf library defaults to using the C++ implementation with bindings. So even if this is a blog post about implementing protobuf in C, it can also help implementations in other languages.

But yes, once you want real high performance, protobuf will disappoint you when you benchmark and find it responsible for all the CPU use. What are the options to reduce parsing overhead? flatbuffers? xdr?

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by TheGuyWhoCodes on Sunday April 25, 2021 @05:52PM (#26937053)
Flatbuffers, Cap'n'Proto and Apache Arrow comes to mind.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by xxpor on Sunday April 25, 2021 @06:51PM (#26937479)
Handwritten ASM for perf is almost never worth it in modern times. C compiled with GCC/Clang will almost always be just as fast or faster. You might use some inline ASM to use a specific instruction if the compiler doesn't support generating it yet (like AVX512 or AES), but even for that there's probably an intrinsic available. You can still inspect the output to make sure it's not doing anything stupid.

Plus it's C so it's infinitely more maintainable and way more portable.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by astrange on Sunday April 25, 2021 @10:15PM (#26938586)
The x86 intrinsics are so hard to read because of terrible Wintel Hungarian naming conventions that I think it’s quite clearer to write your SIMD in assembly. It’s usually easy enough to follow asm if there aren’t complicated memory accesses anyway. The major issue is not having good enough debug info.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by xxpor on Monday April 26, 2021 @12:48AM (#26939326)
I honestly don't think I've seen native windows code in over 20 years at this point. Obviously there's a ton of C++ out there, it's just basically as far away from me as possible.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by ma2rten on Tuesday April 27, 2021 @01:10AM (#26952245)
But this seems to be an edge case where you have to rely on functional programming and experimental compiler flags to get the machine code that you want.

Portability is typically not a big issue, because you can have a fallback C++ implementation.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by pjmlp on Sunday April 25, 2021 @01:41PM (#26934672)
Yet Microsoft was able to make a .NET implementation faster than the current Google's C++ one.

A proof that they don't care enough about protobuf parsing performance.

https://devblogs.microsoft.com/aspnet/grpc-performance-impro...

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by haberman on Sunday April 25, 2021 @01:46PM (#26934733)
From your link:

> Support for Protobuf buffer serialization was a multi-year effort between Microsoft and Google engineers. Changes were spread across multiple repositories.

The implementation being discussed there is the main C# implementation from https://github.com/protocolbuffers/protobuf/tree/master/csha...

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by pjmlp on Sunday April 25, 2021 @02:03PM (#26934934)
I know, the point was that they don't care to improve the C++ version.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by haberman on Sunday April 25, 2021 @02:18PM (#26935110)
The C++ parser has been completely rewritten in the last few years to push the current code generation design to its limits. The biggest obstacle in the way of optimizing it further is legacy APIs, especially std::string for string fields, which forces tons of small heap allocations and frees and defeats the benefits of arena allocation.

Changing very widely used APIs is not easy, these things take time.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by pjmlp on Sunday April 25, 2021 @02:23PM (#26935165)
Interesting, thanks for the clarification, however why do libraries depend on what should be implementation details of generated code to start with?

Something that by good CI/CD practices shouldn't even be part of checked in code.

Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by haberman on Sunday April 25, 2021 @02:30PM (#26935229)
It's exposed in the API. When you access a string field in protobuf, say foo.my_str_field(), the return type is const std::string&. We have to have a real std::string internally to return a reference to it.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by pjmlp on Sunday April 25, 2021 @02:33PM (#26935256)
Thanks for the explanation.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by boulos on Sunday April 25, 2021 @02:14PM (#26935058)
haberman is part of the “they” here (he’s on the core protobuf team, does things like this).
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by barbazoo on Sunday April 25, 2021 @02:56PM (#26935469)
In what kind of scenarios do they use Protobufs? I can think of messaging systems, streaming, RPC, that sort of thing?
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by ch33zer on Sunday April 25, 2021 @03:12PM (#26935635)
Everything at google is built on RPCs between systems using protobufs
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by ma2rten on Sunday April 25, 2021 @03:23PM (#26935719)
Protobuf is also used as a format to write data to disk.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by alephnan on Sunday April 25, 2021 @03:38PM (#26935879)
Why is this being down voted? It's accurate.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by randomswede on Monday April 26, 2021 @05:22AM (#26940735)
And one of the more important things done at the very beginning of each and every HTTP(S) request is packaging the web request up in a protobuf and send it onwards from the loadbalancer.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by barbazoo on Monday April 26, 2021 @12:52PM (#26944769)
So downstream from the LB it's no longer HTTP? Or is every component just protobuf "aware"?
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by h43k3r on Monday April 26, 2021 @02:40PM (#26946295)
Yes
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by TheReveller on Sunday April 25, 2021 @07:13PM (#26937611)
Also extensively for database column data types.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by mkoubaa on Sunday April 25, 2021 @02:49PM (#26935407)
Given that fact I'm wondering if google ever researched custom chips or instruction sets for marshalling pbs, like the TPUs they worked on for ML.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by lupire on Sunday April 25, 2021 @03:29PM (#26935778)
Problem is once you parse the protobuf, you have to immediately do other computations on it in the same process. No one needs to parse protobufs all day long like an ML model or doing hashes for crypto.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by jeffbee on Sunday April 25, 2021 @03:45PM (#26935951)
That doesn't seem to preclude hardware assistance. For example they have also explored hardware acceleration for the tcmalloc fast path allocation by adding circuits to general purpose CPUs. Arguably, Intel BMI descends from a request that Google made years ago to speed up protobuf decoding (and other workloads) on x86.
Re:Parsing Protobuf at 2+GB/S: How I Learned to Love Tail Ca... by jlouis on Monday April 26, 2021 @06:48AM (#26941173)
I'm going to guess most of the time is being spent elsewhere in the systems they are looking at and it is rather rare they have a situation where the parser is dominating. Protobuf is already a winner compared to the JSON mess we're in.