I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, they really don’t work for that use-case—and an ESB means having to try to shove a wire format into pretty much every use-case. So “when it’s bad, it’s really bad” is a bad property for a format aimed specifically at being an enterprise’s ESB-interchange-format dogma to have.
(Contrast to, say, JSON-RPC, which I’d place on the opposite end of the scale being described here. JSON-RPC is mediocre at best in terms of data fidelity, performance, etc., but it’s inoffensive—when it’s bad, it’s no worse than when it’s good. That makes it a good choice of ESB dogma... even though, in engineering terms, it sucks!)
It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original.
As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any implementation of the "right" way to do things is telling.
And, that being said, I think they're right about the conclusion, too. A shorter way to say "Protocol Buffers are a bad choice of common ESB-bus format, even though being an ESB-bus format is the primary thing they're for and what everyone tries to use them for" is "Protocol Buffers are a wrong design." If something doesn't work when used to do the thing it's advertised to do, then it's broken, even if it can do something else.
The only difference between "Protobuffers Are Wrong" and my conclusion is that I'm making the implicit context of their argument explicit. Read between the lines of their argument—they are talking about the use of protocol buffers (specifically, gRPC) in an ESB-bus common-format scenario. None of their arguments make sense if they aren't.
For me protobuf+grpc gives you enough building block, not to shoot yourself in the foot - but rather start from something ready. You get on top of that census metrics, bi-directional communication, and yes there are limitations - it's HTTP2 only (right? correct me if I'm wrong), but maybe that's what you need.
Instead, I see folks - going back to sock(), using them directly, and then reinventing wheels of wheels of wheels.
Then discovery comes - how do I discover what's there and there? I've got used to the nice (forgot the itnernal name) where I can ask the "grpc" server (well stubby, whatever it was called) give me your endpoints - and it gives me the endppoints. Then I can talk to it.
Yes, it complicates build systems (need to generate damn protos), but you can also do custom protobuf generator plugins that generate custom tailored access, where you get performance gains, sacrificing features you don't need, like project perfetto - https://android.googlesource.com/platform/external/perfetto/...
Here: ProtoZero - https://android.googlesource.com/platform/external/perfetto/...
Possibly old, there might be newer version, but still: https://docs.google.com/document/d/1bxlk9F79JZDk4wRXZQ9WQ5DP...
This is the main point where the whole reason to use Protobuf falls apart - it doesn't work well enough to skip manual parsing.
And if you do manual parsing you might well go all the way in and use JSON with a schema and a reasonable library to handle these. You gain very little convenience wise and only a bunch of line bytes at best. Which typically does not matter for use cases of Protobuf that are not Google sized.
- has opinions on interface design
- isn't afraid to be wrong publicly
- is brash on a personal blog
I dunno, this is mostly positive. I'd have to see how well they'd adapt to the much different context and goals of one of our design reviews, but this isn't an immediate red flag. There's a million ways to be bad at a job. I'll give them enough rope to hang themselves in a serious conversation rather than invent the idea that someone would be an unworkable perfectionist professionally just from being a type purist on the internet.
* cap'n proto https://capnproto.org/
* flatbuffers https://google.github.io/flatbuffers/
* hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/
You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beating it, certainly not with protobuf). Seriously, use hdf5 for machine learning and you can restart experiments and ... well they start. They don't spend the first minute or 2-3 reading their data back in.
Protobufs focus on:
* standard (meaning it's perhaps a bad standard but it's standard) (also meaning of course you don't get to choose formats, or perhaps I should say your company will lose a lot if it lets developers choose alternatives to protobuf anywhere) (this is where the frustration comes from)
* streaming support (meaning you can write out and read in protobufs without having to keep the whole thing in memory)
* extendable (including sort-of kind-of backwards-forward-compatibility. Meaning old code can read in a new version protobuf, change something and write out a proto that still has the fields it didn't understand)
* language agnostic
* composability (meaning cat protobuf1 protobuf2 > resultbuf means resultbuf is deserializable)
* an attempt to efficiently store integers (meaning it's integer encoding format is bloody complex, but "space efficient", except not quite so efficient it doesn't need compression at which point, why bother ?)
And in this protobuf will beat the above formats (except cap'n proto).
None of the alternatives to protobuf existed when it was invented though. The only alternative in existence was ASN.1. And it beat that, by a LOT.
Mainly, I just think that interchange formats should be boring, simple, and ideally not incredibly slow. Protocol buffers at least meets those bars, even if it doesn't meet the consistency bar that the article desires. The whole "stop being a hipster and just use X" meme comes to mind. Probably realistically X is JSON in this day and age.
The problem with JSON for internal formats is that there’s often only one consumer and producer, so documenting the format rarely happens. Later, when you want to reimplement one side, you learn that there is no “one place” where you parse the JSON, but that you hand bits and pieces of it to completely unrelated areas of code.
Figuring out these as hoc formats is nigh impossible, so you end up just looking at the data over the wire and writing code that parses what you see. Except then you have broken code that doesn’t parse the less-common variants that have extra fields, don’t have extra fields, sometimes have their fields as stringified ints instead of just binary ints, etc.
Virtually every time I’ve seen JSON used as an internal interchange format, reverse engineering that format a few years later.has become a massive, error-prone, tedious, and time-consuming task. Save yourself from this ahead of time and pick formats that require a predefined structure, like protobufs.
The one place where I don’t think dynamic types have their place is in internal interchange formats. These can live forever, and they can change subtly over time. It’s the living forever part that changes the calculus.
Presumably you would be okay if the care and rigor were taken to define a schema and its physical representation just happened to be JSON?
How many of those people exist?
Don't forget integer type defined by implementation. Very easy to shoot yourself in the foot in when consumers/producers are not written in the same language, even more if it's a mix of dynamic and static languages.
Nothing beats a well defined and commented schema.
When my team was deciding on a binary serialization and RPC communication format, we needed something that ran on a Cortext m3 with no malloc, and every major phone platform (including WP at the time) and desktop OS.
It came down to cap'n proto, and Protobufs.
Protobufs runs everywhere. It won.
Protobuf has disadvantages, but mapping types are something I frequently make use of. losing out on them is a big deal
That said, I think maps are a fine feature for a serialization format to have. But, I haven't gotten around to adding them in Cap'n Proto because I have yet to hit a use case where I wasn't happy with a list of key/value pairs.
It follows sorta the same reason that Clojure became a fairly widely used lisp – mapping types are extremely common in real world programs. Having them supported first class makes a lot of common tasks quicker to get through.
What does bother me is that in the above generic type, "key" and "value" must be pointer types (i.e. non-scalar). The reasons are not unreasonable but it's definitely obnoxious in practice.
Parquet is gaining a lot of traction lately for this use case.
That said...I used protobufs to encode the log of Tungsten Replicator, a replicator for MySQL. We did not have a single version related error while I worked on it over a period of years. That was essential given the fact that replication logs are an on-disk format that must remain stable across version upgrades. Our logs in aggregate contained billions of transactions--you can't not read an older log file just because you are running a new software version. The claim that protobufs do not handle versioning is nonsense.
Speaking of which, the generated code was incredibly convoluted with what seemed like tangled dependencies on the underlying protobufs library. I used to dread debugging it--sometimes I would have to go into it to figure out what the upper layers were doing wrong. Luckily it did not happen very often. And I don't recall ever hitting a bug in protobufs itself for all that the code was virtually unreadable.
Most likely, the vast majority of Protobuf use is with code generated to a single, canonical location (which leads to the problem described above).
But hypothetically, it would be possible for code to generate and refer to two different versions. That doesn't sound impossible. Maybe just improbably given the way things are typically set up.
> They're clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has.
This dovetails with other arguments that I've seen recently that are becoming more frequent:
Have we entered a new world where the lessons of companies working at massive scales are not only generally superfluous for smaller scales, but are actively harmful?
I think so, yeah.
Microservices turn out to have a lot of negative consequences, and their positives work best when you have dozens or hundreds of developers. If you've got a handful of developers... not so great.
Whenever I see large numbers of microservices anywhere my null hypothesis is that some organizational disfunction is leading teams to factor applications into unnecessarily small pieces.
You can take your logic and apply it to anything.
For example killing innocent babies:
"Killing babies has constraints, advantage and disadvantages. No other action in the universe is free from having advantages and disadvantages. They all have different use cases...
I don't get why the author is 'spouting dogma' about how killing innocent babies is bad.
Be wary of people who argue about killing babies etc etc."
This relativism you're expressing can be seen as ridiculous when applied to killing babies - it can actually seem to make sense when you use it to justify your own biases - which is what you're doing. Of course doing that has it's advantages and disadvantages, etc etc no opinion is better or worse everything is relative :)
Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages.
No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases.
If you are building a system where your data can be described by protobufs, it may be a good choice. If your data structures don’t mash up well and you have to manipulate them heavily, protobufs may be a bad choice.
Instead of pointing out use cases where a different serialization format may be better than protobufs and use cases where protobufs are better, and why, the author is spouting dogma about how protobufs are bad for every use case.
Be wary of working with developers who prefer to argue about why they hate certain technologies instead of providing useful data and ways to solve problems. You don’t always have to solve a problem just because you are aware of it but don’t go shouting from the rooftops that a technology sucks for every use case under the planet when that’s obviously not the case. The author’s opinion is more of: I don’t like protobufs. Not: protobufs are wrong