Add extra stuff to a “standard” encoding? Sure, why not

Re:Add extra stuff to a “standard” encoding? Sure, why not by NavinF on Wednesday September 20, 2023 @03:21AM (#37581365)

> didn’t find any standard for separating protobuf messages

The fact that protobufs are not self-delimiting is an endless source of frustration, but I know of 2 standards for doing this:

- SerializeDelimited* is part of the protobuf library: https://github.com/protocolbuffers/protobuf/blob/main/src/go...

- Riegeli is "a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding": https://github.com/google/riegeli

[ Parent | on HN ]

Re:Add extra stuff to a “standard” encoding? Sure, why not by harvie on Wednesday September 20, 2023 @03:57AM (#37581615)

https://msgpack.org/

[ Parent | on HN ]

Re:Add extra stuff to a “standard” encoding? Sure, why not by vore on Wednesday September 20, 2023 @05:25AM (#37582179)

Huh? This is an entirely different serialization format altogether.

[ Parent | on HN ]

Re:Add extra stuff to a “standard” encoding? Sure, why not by formerly_proven on Wednesday September 20, 2023 @06:37AM (#37582559)

Just stick this in front of your frames:

    81 A2 70 62 C6 <32 bit be length> <protobuf data>

[ Parent | on HN ]

Re:Add extra stuff to a “standard” encoding? Sure, why not by KMag on Wednesday September 20, 2023 @11:36AM (#37585544)

You got me. What's that format / magic number? At first I thought it was the zlib header for incompressible data, but that's not right. Is that just the MAC address of some NIC you own?

[ Parent | on HN ]

Re:Add extra stuff to a “standard” encoding? Sure, why not by aidenn0 on Wednesday September 20, 2023 @02:02PM (#37587545)

It's msgpack:

0x8n A map containing "n" key/value pairs -> 0x81 introduces a single key/value pair

b101XXXXX : A string where XXXXX is the string length -> 0xA2 introudces a 2 byte string

0xC6 followed by a 32-bit value N introduces an array of bytes of size N

0x70 0x62 "pb"

So it roughly corresponds to

  {"pb": <protobuf data> }

[ Parent | on HN ]

Re:Add extra stuff to a “standard” encoding? Sure, why not by paulddraper on Wednesday September 20, 2023 @08:11AM (#37583207)

Poster is being cute.

[ Parent | on HN ]

Re:Add extra stuff to a “standard” encoding? Sure, why not by twic on Wednesday September 20, 2023 @10:45AM (#37584843)

Create a MessagePack message containing a byte array, put the protobuf in that. Of course, the enterprise solution would be to hex encode it and put it in a CDATA section in an XML document.

[ Parent | on HN ]

Re:Add extra stuff to a “standard” encoding? Sure, why not by retSava on Wednesday September 20, 2023 @08:55AM (#37583536)

Doesn't have a Brainf*k implementation, disqualified.

[ Parent | on HN ]