> There's not a lot of other options.

There's one I like the most, Microsoft .NET Binary XML. Here's a decade-old overview: http://const.me/articles/net-tcp/

Has unlimited extensibility and doesn't need schema /protocol to decode. May include typed values not just strings from XML, makes serializers/deserializers much faster compared to text/xml, no need to print or parse numbers. Has easy way to convert to text/xml, helps with debugging and diagnostics. All protocol elements are byte-aligned, with some experience it's possible to understand even without decoding into text.

The only downside, hard to implement outside .NET. Still, in 2020 all editions of .NET runtime support that thing, and these implementations are compatible across platforms.

Don't miss some of the commentary below about zero-copy encodings.

Also, the ITU-T specs for ASN.1 are some of the best-written specs I've ever had to read and use. ASN.1 may seem dated and ugly, but its world-class specs are a real advantage, IMO.

I've written zero-copy implementations (as much as was possible) for DNS, MIME, RTSP/RTP, SMTP, and others. Locality is far more important to performance. Generic zero-copy APIs often have too much indirection, and for short messages or messages with many distinct fields, generating all those pointers can cause more data churn than simple copies. And while zero-copy makes the most sense for messages with large blobs, if you have to buffer the entire message you're again losing data locality, whereas a streaming architecture can be even more performant even with copies (and even because of the copying) when you can keep your working set small and hot.

IME, there's simply no getting around the fact that if you want better performance, efficiency, and/or latency, you have to design your entire stack holistically. A library promising a generic API with a supposedly generic "optimization" is often already suboptimal, and architecting an application with that mindset (e.g. trying to push all the hard parts behind a generic getter/setter API) makes the suboptimality grow non-linearly.

I agree about ASN.1. The ASN.1 specifications are well written and everything is very well thought out (even though it's not always obvious why without the benefit of hindsight). Though, for a long while they weren't freely available, officially or unofficially. Thankfully that's changed (not sure precisely when that happened), but it cost them a lot in terms of mindshare and was a big reason, I suspect, for the horrible open source tooling both then and now.

Just gonna add this here: https://news.ycombinator.com/item?id=23589117 (mostly for the followup discussion, see for example https://news.ycombinator.com/item?id=23589357).

There's tradeoffs. "Packed encoding type is generally superior" for some workflows (write-once serialization across the wire), but certainly not across the board.

There are tradeoffs, but I think binary, non-TLV encodings are superior. TLV encodings are notoriously difficult to encode but add no real value on the decode side. With TLV encodings you get a measure of implied extensibility that requires explicit support in binary, non-TLV encodings. But it turns out that you want to have explicit extensibility even in TLV cases anyways, since it helps the code generator, and for security reasons.

Even ignoring the zero-copy advantages of flatbufs and Cap'n Proto, TLV is bloaty and redundant, and redundancy is one mother of serious bugs in hand-rolled codecs.

Perhaps the best example I have of trade-offs is PostgreSQL's JSONB, which is optimized to be compressible and fast to traverse. To my knowledge that's the only binary JSON encoding that does that. (Specifically, it encodes lengths of array and object elements, and of object keys, but every N it encodes an offset, and the lengths are of the value encodings which are themselves concatenated. This allows similar lengths to be compressible, while the never-repeating offsets are not but also they're few and far between, and it also allows for fast indexing.)

> TLV is bloaty and redundant, and redundancy is one mother of serious bugs in hand-rolled codecs.

This is an argument against hand-generating TLV binary packets, not an argument against using generated code (as in protobuf).

> Even ignoring the zero-copy advantages of flatbufs and Cap'n Proto

See kenton's comment I linked, the zero-copy can be a disadvantage in write-many applications, since if you have a variable-length field (a string or submessage) and you can end up with super inefficient binary encodings.

> zero-copy can be a disadvantage in write-many applications

To clarify, the disadvantage stems from the fact that to get zero-copy, you really need all the parts of a particular message to be in contiguous memory (or at least a small number of contiguous segments), which creates a memory allocator design challenge: now instead of one huge heap you have many small heaps and fragmentation potentially wastes space on the wire. So far I've made no attempt to solve this problem in Cap'n Proto; new objects are simply always added to the end of the message, which works fine for write-once and append-only scenarios but not for general mutable structures.

I think this is an orthogonal issue to the TLV vs. fixed offsets debate.

Right, this is a problem, but it's orthogonal to TLV, but it's TLV I was arguing against, not necessarily for cap'n proto.

You can use OER without having to allocate all sub-parts of the structure from one flat buffer, with no "edits" that fragment the buffer, and no memmoves, and you won't get zero-copy, but you'll still get a more efficient codec than any TLV encoding. I just don't see any case where TLV is appealing, but TLV is appalling :)

I think we're on the same page.

EDIT: BTW, thanks for the clarification -- I thought that was the issue you meant, but what you linked to was lengthy and I didn't want to have to wade through all of it.

EDIT: Also, even if zero-copy schemes are inefficient for some uses on the encoder side, they can still be very efficient on the decode side, as you need essentially no memory allocations to decode (though you still need to traverse the encoded data).

> I think this is an orthogonal issue to the TLV vs. fixed offsets debate

I agree, I don't quite think they're related. But I think some of the original claim that tlv is worse is based on the potential for zero copy. That is advantageous in some common situations, but it's not always the case.

> This is an argument against hand-generating TLV binary packets, not an argument against using generated code (as in protobuf).

Or maybe it's an argument against the kind of designs that tempt devs to hand-code them.

> See kenton's comment I linked, the zero-copy can be a disadvantage in write-many applications, since if you have a variable-length field (a string or submessage) and you can end up with super inefficient binary encodings.

I don't buy that as a counter argument to the point about TLV's disadvantages relative to PER/OER and flatbuffers/cap'n proto. You might prefer OER to cap'n proto in some cases, or vice versa, but not TLV to either.

What I do appreciate about TLV-style formats is general-purpose debugging tools. When all you have is a bitstream where field lengths are not multiples of 8 bits and are variable based on the actual value being encoded (looking at you, MPEG), the toolset to debug streams ends up being non-trivial.

If the IDL compiler is more like a library then you can have an "asn1print" that takes a schema and dumps the data correctly with all the metadata you need and want and none of the crappy TLV.

Haven’t used protobufs for years but as I remember it you can use an old schema to read a newer version object, discarding the new fields. I don’t think that would be possible without the TLV.

This is called "extensibility".

In ASN.1 people were doing the same thing unofficially in the mid-80s, then explicit support was added in the form of "extensibility markers".

There's nothing new in protobufs.

EDIT: I should add that the non-TLV encoding rules for ASN.1 all support extensibility. Also, there are good engineering practices and security reasons why one should want explicit documentation of what structures/enums/choices/typeranges are expected to be extensible. Relying on decoders to ignore unexpected bits is not a sound practice.

flatbuffers also support this, but use offset tables instead of TLV.

It may be that TLV is necessary if you want to do this in a format that is designed for streaming rather than random access.

Nice summary!! :)

Thanks!