I’m not sure how unpopular this opinion is, but I’d actually argue that there are underrated human benefits to using “just protobuf”, which are not shared by using “just JSON”.

1. Single source of truth for certain types, enums.

2. (assuming you’re using gRPC too) A separate, minimalist, and language-agnostic definition of your service interface, which you can document with comments to your heart’s content, and which (unlike other documentation) can NOT be out of sync with the actual service. I don’t have to read C++ to understand your C++ service should work.

3. Protobufs encourage message type / enum reuse, by allowing you to import other definitions. This might seem trivial, but it’s super important in mediumish orgs that everyone is using the same definition of time, geography, etc. It all adds up to less surprises when you open up a new .proto file.

The kicker is not that you can’t somehow get these things with JSON-over-HTTP, too. It’s that protobufs-over-gRPC won’t work without them. The trade off is that you can’t inspect raw requests unless you have built some tooling around it.

I think the missing assumption is that doing "just protobuf" is not always a possibility when you in the end have an open API front-facing the internet.

So now there is a point in your codebase where you are receiving data in a JSON form. At this point having protobuf elsewhere is not chosing between JSON and protobuf, it's chosing between "json+protobuf" and "json".

Not really. If you are sane you carefully parse all data from the internet and sanitize it first thing. json+protobuf is in fact an advantage because by keeping your internal and external data formats different you eliminate one class of mistakes where external data makes into internal systems. (this isn't perfect, you can change format without doing sanity checking but it makes it harder to do accidentally, and easier to find in an audit)

In addition to what my sibling comment says about "sanitizing" external data – which I wholeheartedly agree with – the "open API front-facing the internet" is in the process of changing, if you're talking about normal websites.

gRPC-web isn't quite feature complete relative to normal gRPC, but it is getting pretty close, and the gains of avoiding JSON (de-)serialization would be big. I think once the protobuf story has a complete chapter for the front-end, bigger engineering orgs will roll it out much like they're rolling out typescript today.

If you're talking about actual developer/public-facing APIs, those will probably remain in JSON land for a while.

#1 and #3 can be solved by using TypeScript types to define the schema of your JSON.

IMO they're readable enough to serve #2 too.

Before someone mentions "JSON schema" - I've tried to use them, and they're difficult to write and impossible to read. Reading/writing TypeScript types are a dream in comparison.

I find that the human benefits of using something like protobuf outweigh the human benefits of using JSON. The machine benefits are just a cherry on top.

Protobuf - or, more specifically, proto files - gives you a central place where you can define and also document your formats. You can throw an ASCII-field UML sequence diagram in there, if you need to. And it's right there in the single file that everyone will use to communicate the protocol, and the protocol at least can't change in any structural way without editing that file, so it's got a much higher chance of being kept up-to-date, and of being read by the people who need to read it, than any of the available options for documenting JSON-based protocols.

All JSON gives you is human readability, and browsers can read it without a library. The 2nd, I don't care about with back-end services. The 1st I don't really care about at all, because command-line utilities and library functions for dumping protobuf datagrams to a text format are a dime a dozen.

You don't even have to get close to Google or FB before running into issues with JSON. At a previous job we were working on REST APIs that topped out at around 100qps. Performance wasn't what we wanted and we were initially pointing the finger at Python and the database. We started profiling the application and found that most time was spent in JSON serialisation - which is actually implemented in C. Unless you're serving the browser there's really no excuse for not using a binary format.

The excuse is that there are not enough seasoned software developers, and what used to be a relatively meritocratic discipline is now a shit show.

Problems started somewhere in early '00s when "software architect" became a dirty word, and one hit wonders like Paul Graham pontificated that "young is smarter".

And here you are.

I've never seen JSON used in a project of any size at all where we didn't end up trying to hack constraints and types on top of it with some ugly and gross thing or another—looking at you, JSON Schema.

The more parts of your stack you tighten up, the fewer errors you'll hit and the more flexibility you'll have to use less-strict tools when it really matters. That's true at all but maybe the smallest scale. Worrying about the costs of having to document what you intend in a way your machines can verify—I mean, shouldn't you be doing that anyway?—is baffling to me. You don't need anything like Google scale to see the benefits of it. It's basic communication AFAI am concerned.

This is not a binary, though. E.g. Thrift has multiple transport/protocol settings, such that you can use JSON over HTTP for debug/testing purposes while still using a nice, compact binary representation over a raw TCP socket for production.

They both exist for a reason. JSON is really nice for your quick proof-of-concept where you have relatively few developers working on a small system. Up to a point, this also scales well.

On the other hand, if you have many developers, and the problem space starts converging toward scale, then certainly the problem space is different. Encoding of data cannot follow the language with the weakest type zoo, and efficiency starts to matter.

The hard part is to know when to make the switch, or when to anticipate growth in advance, such that you pick the right tool for the job.

It isn't just a question of the machine. For complex messaging, the added value of a well-defined message typing, provided by protobuf, will help. It will also remove a lot of problems if you have multiple different languages in the stack, talking to each other.

>JSON is really nice for your quick proof-of-concept where you have relatively few developers working on a small system.

This is really, really bad thinking and will always cause you a timebomb that will 100% explode on you in the long run.

"just a quick POC" will always end up being the actual product, and once you start down this path you end up with "let's just use mongo, it's schema-less, it'll be faster for devs and we can use a proper db later"... "let's just use nodejs for the POC, all the devs already know JS so it'll be great"... a few years later you end up with some gargantuan monstrosity that nobody wants to maintain, your "quick" language/db-du-jour is dead and unmaintained, you're EOL on four different software fronts and you now get to explain to the bosses why you need to rewrite the last four years of dev work.

Once you hit the wall with JSON what format do you use? At a past job we were stuck using SOAP/XML middleware. We hit walls but I’m not sure if a different format would work. We ended up rate limiting lower priority traffic to get past our issues.

Protobuf/gRPC.

CSV.

Just kidding, but I suspect it would be a lot faster.