What a pleasant surprise it will be for you when you find out that jq silently corrupts integers with more than 53 bits.
But still, given that easy consumption from JavaScript is the ultimate primordial reason for choosing JSON over other formats, it seems like trying to transmit integers with more than 53 bits of precision over JSON is asking for trouble. Because it's only a matter of time until someone will want to do something like write a new service in Node, and the JavaScript parsers for other formats are at least somewhat more likely to guide people toward using BigInt for large integers.
[1]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
I explicitly asked for and achieved step 2. in the modified SerializeJSONProperty algorithm[1] so that users could decide and opt-in to serializing BigInts as strings if they so choose, with or without some sigil that could be interpreted by a reviver function. e.g.:
> JSON.stringify(BigInt(1))
TypeError: Do not know how to serialize a BigInt
...
> BigInt.prototype.toJSON = function() { return this.toString(); }
> JSON.stringify(BigInt(1))
'"1"'
No, it's typically IDs. And my observation is that inexperienced devs start out serializing 64bit IDs as ints and get burned first before they move to text.
There are several reasons why you end up with IDs > 2^53 even if 2^53 is good for enumerating ~100k things per earthling:
Many cloud providers will give you IDs that are specified to be 64 bit uints, and you'd better not corrupt some fraction of them. Generating unique integers strictly serially does not scale, so you might end up with something like Twitter's snowflake. Or you might want to tag additional info (reserve a few of the low bits for the type of id, or reserve particular ranges for particular types of acccounts etc etc).
I can promise you I have run into the issue in real life more than once and it's not just javascript and jq that like to silently corrupt ids by truncating them to double precision. For example pandas is really great at it as well.
What makes it especially fun that often only a very small fraction of IDs will be affected (< ~1 in 2000 if uniformly distributed).
The fact that JS has 53 bit precision will be a JS problem whether you use protobuf or anything else. On the other hand, if you are not using JS, it will almost certainly parse numbers of the precision that your language offers.
That's impossible with text-based formats like XML and JSON, because everything is fractally variable-length.
At smaller scales, I think the (machine) benefits of using something like protobuf don't nearly outweigh the human benefits of just using JSON.
1. Single source of truth for certain types, enums.
2. (assuming you’re using gRPC too) A separate, minimalist, and language-agnostic definition of your service interface, which you can document with comments to your heart’s content, and which (unlike other documentation) can NOT be out of sync with the actual service. I don’t have to read C++ to understand your C++ service should work.
3. Protobufs encourage message type / enum reuse, by allowing you to import other definitions. This might seem trivial, but it’s super important in mediumish orgs that everyone is using the same definition of time, geography, etc. It all adds up to less surprises when you open up a new .proto file.
The kicker is not that you can’t somehow get these things with JSON-over-HTTP, too. It’s that protobufs-over-gRPC won’t work without them. The trade off is that you can’t inspect raw requests unless you have built some tooling around it.
So now there is a point in your codebase where you are receiving data in a JSON form. At this point having protobuf elsewhere is not chosing between JSON and protobuf, it's chosing between "json+protobuf" and "json".
gRPC-web isn't quite feature complete relative to normal gRPC, but it is getting pretty close, and the gains of avoiding JSON (de-)serialization would be big. I think once the protobuf story has a complete chapter for the front-end, bigger engineering orgs will roll it out much like they're rolling out typescript today.
If you're talking about actual developer/public-facing APIs, those will probably remain in JSON land for a while.
IMO they're readable enough to serve #2 too.
Before someone mentions "JSON schema" - I've tried to use them, and they're difficult to write and impossible to read. Reading/writing TypeScript types are a dream in comparison.
Protobuf - or, more specifically, proto files - gives you a central place where you can define and also document your formats. You can throw an ASCII-field UML sequence diagram in there, if you need to. And it's right there in the single file that everyone will use to communicate the protocol, and the protocol at least can't change in any structural way without editing that file, so it's got a much higher chance of being kept up-to-date, and of being read by the people who need to read it, than any of the available options for documenting JSON-based protocols.
All JSON gives you is human readability, and browsers can read it without a library. The 2nd, I don't care about with back-end services. The 1st I don't really care about at all, because command-line utilities and library functions for dumping protobuf datagrams to a text format are a dime a dozen.
Problems started somewhere in early '00s when "software architect" became a dirty word, and one hit wonders like Paul Graham pontificated that "young is smarter".
And here you are.
The more parts of your stack you tighten up, the fewer errors you'll hit and the more flexibility you'll have to use less-strict tools when it really matters. That's true at all but maybe the smallest scale. Worrying about the costs of having to document what you intend in a way your machines can verify—I mean, shouldn't you be doing that anyway?—is baffling to me. You don't need anything like Google scale to see the benefits of it. It's basic communication AFAI am concerned.
On the other hand, if you have many developers, and the problem space starts converging toward scale, then certainly the problem space is different. Encoding of data cannot follow the language with the weakest type zoo, and efficiency starts to matter.
The hard part is to know when to make the switch, or when to anticipate growth in advance, such that you pick the right tool for the job.
It isn't just a question of the machine. For complex messaging, the added value of a well-defined message typing, provided by protobuf, will help. It will also remove a lot of problems if you have multiple different languages in the stack, talking to each other.
This is really, really bad thinking and will always cause you a timebomb that will 100% explode on you in the long run.
"just a quick POC" will always end up being the actual product, and once you start down this path you end up with "let's just use mongo, it's schema-less, it'll be faster for devs and we can use a proper db later"... "let's just use nodejs for the POC, all the devs already know JS so it'll be great"... a few years later you end up with some gargantuan monstrosity that nobody wants to maintain, your "quick" language/db-du-jour is dead and unmaintained, you're EOL on four different software fronts and you now get to explain to the bosses why you need to rewrite the last four years of dev work.
Just kidding, but I suspect it would be a lot faster.
This specification allows implementations to set limits on the range and precision of numbers accepted.
Other implementations can have other limits, but that does not mean that JSON itself or all implementations have the specific limits of javascript.
>JSON.stringify(1n)
TypeError: BigInt value can't be serialized in JSON
>JSON.stringify(1n)
Uncaught TypeError: Do not know how to serialize a BigInt
at JSON.stringify (<anonymous>)
at <anonymous>:1:6
Javascript can do arbitrary precision integers now. So the last part is not true. You can generate your JSON strings with string concatenation in JS if you like.
Just because JSON.stringify has some limit doesn't mean JS has.
For everything else, it is really a non-issue.
> 9007199254740991n * 9007199254740991n
> 81129638414606663681390495662081nBecause learning how to use correct tool for the job is so difficult and so outside of dev/qa job descriptions, that it's much better to waste compute on your servers and deal with performance and scalability problems.
Your comment is unnecessarily inflammatory.
> AWS bills can easily eclipse engineer salaries if you use it inefficiently.
If this happens, it is more due to architectural and algorithms problems than to the runtime of the language. There are many large applications written in Python, Ruby and PHP that run just fine.
It's a pain, but it's a short term pain, and necessary.
Is your turnover so high that learning a single tool is a significant portion of employee costs?
I think recent experience should tell us that the easiest thing plainly wins out. JSON can be terrible, Javascript can be terrible, but limitations on large integers isn't the worst hurdle to take.
"if you only need 53 bits of your 64 bit numbers"
JSON numbers are arbitrary precision.
"blowing CPU on ridiculously inefficient marshaling and unmarshaling steps"
On the other hand I am not blowing dev and qa time on learning/developing tools to replace curl/jq/browser/text editor.