Welcome to the world where everyone is wasting time and space on JSON encoding rather than using some sensible serialization protocol that can handle raw bytes.

Without any explicit admission of guilt on my part; what are some good options here? Protobuf is cool but I really don't want to mess with a special compiler and all that.

Common alternatives are Apache Avro, MessagePack, FlatBuffers, and Thrift.

I have no idea which is best. Frankly it seems like there are too many choices.

Probably MessagePack, since it is JSON-like and I've actually heard of it.

Avro has some really cool features like inbuilt schemas, schema versioning and migration (e.g. deprecating or renaming fields) but you pay for them with more overhead than MessagePack.

Protocol Buffers have schemas too (though versioning and ensuring compatibility is quite messy and requires understanding internals). And it has less overhead than MesssagePack.

I'm not sure what Avro is doing, but as a rule schema enables you to have less overhead, rather than more. The main advantage of MessagePack over schema-based formats is that it's dead-simple and mostly compatible with JSON. Schema-based formats usually need either a code generator or maintaining an annotated version of your data classes and making sure they match the schema.

(Of course, with JSON or MessagePack you might still end up using a serialization library and something like JSON Schema).

Oh, as I understand it Avro's schemas aren't just "built in" as in supported as a first-class part of the protocol, but rather that each message includes with it the schema needed to interpret it. This adds overhead to every message (it's still a binary protocol, though) but crucially it avoids a whole category of hassles around schema distribution and updating.

CBOR could also be a good fit

I just can't get over Cabo's choice to put 65 (yes 65) Bit fixed sized ints in the wire format and to top it off by making it ones complement (the sign bit is part of the type and the longest fixed sized ints are 64 bit range)...

Not familiar with this at all, but I assume it's so that one type can encode and store mixed lists of 64-bit signed and unsigned integers.

HTTP multi-part. It’s a decades-solved problem and works with whatever the current request compression scheme is (gzip, brotli, etc).

It’s not even an HTTP invention, it’s RFC2046 MIME from 1996. RFC2388 standardised its use in HTTP in 1998.

The elegant thing about MIME is it allows multiple encodings and cross-references, so you can have your HTML and the images displayed in the HTML both optimally encoded in the same document, which was handy back in the time when HTML emails were taking off and marketing insisted that the fancy image signature they designed had to show every time, even when the person was reading the email offline…

Of course back then we had to encode the image in base64 anyway because of non-8-bit-clean email servers. But I digress and will go back to my rocking chair.

The responses you got I think literally answer your question but probably aren't what you're going to reach for any kind of HTTP based thing. Your go-to should probably be multipart/form-data. It's well supported in every language and HTTP library and you can send both JSON and the file in the same payload.

There seems to be a common trend of people writing "JSON APIs" thinking that every other part of HTTP is off-limits.

If you're looking for the simplest possible replacements, MessagePack (for data in transit) and SQLite (for data at rest) are worth considering

Wrong hill to die on IMO. I'm saving myself and others a lot of work by using a format that every modern language understands, without any external dependencies. It matters to me because I do a lot of integrations and if every service used their own bespoke format I would go mad.

And the wire size is not that be - first, because most messages are small anyway (unless somebody is doing something stupid like sending files via json), and second, because HTTP compression is there and it works great for text formats like JSON.

> unless somebody is doing something stupid like sending files via json

Like mp3s?

I upvoted you solely for using "first / second" instead of "firstly / secondly", like most people stupidly do these days.

So, you're adding overhead of compression, decompression, parsing and serializing JSON at every step. All likely backed by a language where computing length of a string is O(n).

And people are surprised software keeps being slow despite increase in compute resources. This is insane.

There is an alternate universe where you comment how insane it is engineers waste time on trying to understand the nuances of every single bespoke byte encoding/decoding technique used between services. JSON is just fine for most tasks, otherwise the world would be in flames right now

I mean the world is kind of in flames, if planes were as unreliable as most software is we'd have a hundred falling out of the skies every day. But the only reason it isn't worse, and I can't remember who the quote is from, is that the real Moore's law is the fact that talented hardware engineers keep improving hardware more quickly than software is getting slower.

There's an alternative world where everyone is performance oriented and those Rabbit devices don't run for five hours but for five days on modern batteries. Let's be real the thing is basically a Tamagochi, it should cost 50 bucks and run on chips from 2010.

> it should cost 50 bucks and run on chips from 2010.

If we're really honest it should run on Tamagotchi-era hardware.

We have much more compact and widespread generally available formats out there with libraries for many languages. Want unbounded JSON but more compact? msgpack or bson. Want stuff more efficiently packed based on the message structure? Use protobuf.

Yes, there is a little more effort needed for the engineers there. But ya know, if the inputs and outputs of a thing are actually DOCUMENTED and the schemas are available, it's not some massive reverse engineering feat :P

(Okay, maybe you're stuck doing something in a niche environment where handy protocol/format libraries aren't available to you; MATLAB for Microcontrollers or something. But if you're there, you're probably having fun dealing with all the nuances of implementing an efficient and safe recursive unicode text serialiser/serialiser for a format line JSON anyway :P)

Also if you're going the fairly standard route of "web API over HTTP", the protocols give us way more options readily available for much more efficient streaming of binary data.

It's not "wasting time" to teach devs that there are better ways of doing stuff. base64 encoding mp3s into JSON strings strikes me as "junior dev given 2 weeks to quickly implement something without somebody there to review and suggest alternative ways of doing stuff".

> msgpack or bson. Want stuff more efficiently packed based on the message structure? Use protobuf

Everything you listed is an external dependency in Javascript/Python, base64 is baked in, everyone gets it, everyone's done it.

If you want better interchange you should be pushing it to be included along b64 at the language level not trying to get every dev to include extra dependencies at either side of the exchange.

> If you want better interchange you should be pushing it to be included along b64 at the language level not trying to get every dev to include extra dependencies at either side of the exchange.

Oh, absolutely. JavaScript Object Notation became a defacto standard purely because js could parse it natively. Then `json` was adopted as part of the standard python libraries, within PHP, etc... Once upon a time, even stuff like base64 encoding/decoding required someone to write the code for it. Agreed, it requires pushing to get useful stuff into "batteries included" stdlibs.

We're using JavaScript Object Notation because.... Isn't the name quite telling? :-)

MP3 tags are not there just to being pretty.

Also, if any, an external JSON file should be parsed, not the files themselves.

In my country, no one of these 'self-called engineers' wouldn't even earn a trade/vocational degree. Legally they wouldn't even be engineers.

Use Cap'n'proto and get encoding efficiency and a convenient dev experience. It continually amazes me that devs choose to live in a world of pain.

CBOR is basically JSON but with binary keys and values

> unless somebody is doing something stupid like sending files via json

But that’s exactly the complaint the GP was making here: That mp3s are being base64 encoded rather than using a transmission protocol that handles raw bytes.

So despite your counter argument, you actually agree with the GP.

They will not understand your point. It is a kind of invincible ignorance, a little like how rich kids can’t understand why anyone has to budget.

We’ve squandered almost all of the advancements.

I get what your saying but it’s what everyone uses. At this point it’s like saying write assembly instead of C.

Hopefully people come around but I doubt it.