JSON is unstructured. Every one who parses the data needs to write a parser. As it’s unstructured how do you manage schema changes?
Using a binary format like avro you get strong typing (basic validation), schema change modes, no need to write clients. You can publish your avro schemas to a registry and you then get a catalog of data available anyone can browse to see what data is available and how it looks.
> This just smells like tech debt I’d say the opposite. Using JSON is tech debt. Everyone writing parsers/serialisers, not knowing what fields are available, what values the fields can accept, how to handle new fields / deprecating old fields.
Debugging is simple, AVRO can be inspected unlike over formats with its Generic record format that can also be represented and dumped as JSON.
people should use binary strongly typed formats more. It makes things so much easier for the developer,data evolution and data processing. You also get benefits that your application is no longer running in debug mode.
Gzip doesn't buy much advantage on small records, because there's not much repetition in the source document. And there's no law against compressing binaries, last I checked.
> Next, I have yet to see somewhere... anywhere, not at the scale of something like Uber, or a high-performance game server, really need to worry about serialization and deserialization of JSON.
I built a discrete-event simulator[0] which presents its findings with a simple web interface from the localhost. It's about the simplest possible setup. The simulator can process millions of simulated events in a few seconds, but it takes much longer to stream the JSON to the browser for rendering. Even with "GZIP, baby".
As an experiment I converted it to stream records in Avro format. There are several javascript libraries and Golang libraries. The conversion took me an afternoon. The amount of data streamed went down by approximately half. I didn't gain on rendering time because it currently completes the entire stream before conversion. If that wasn't the case I expect the Avro version would noticeably faster.
So if "it took one random person an afternoon to hook up some libraries" is tech debt, then I guess I better call my bank, because I do that pretty much every day.
Also using JSON in a statically-typed language is tedious and error-prone.
Next, I have yet to see somewhere... anywhere, not at the scale of something like Uber, or a high-performance game server, really need to worry about serialization and deserialization of JSON. Almost all serialization libraries are written in C for MAXIMUM PERFORMANCE regardless of the language you're using, or some sort of hand-tuned-magical bytecode fuckery which has the same effect.
I honestly can't imagine this being useful except for obscenely high throughput service to service communication or as an internal serialization protocol (for something like FoundationDB where you're storing the Avro encoded instance to disk). Truly, JSON and REST are good enough for 99.999% of services and the overhead of using something non-standard will likely never pay off due to the increased developer overhead and maintenance.
This just smells like tech debt and an architecture that can't be debugged.