MessagePack: like JSON, but fast and small

Re:MessagePack: like JSON, but fast and small by neuland on Tuesday March 10, 2020 @02:39PM (#22538196)

Yes, and that's what I did. In my case, I was writing to a file. And my languages were C, Python, and Lua. So, I wrote a small package (plus bindings for Python and Lua) that would write out the message length as a uint64, then a uint8 for which schema the message is (I was working with multiple different message types), and then the actual message bytes.

So, it's do-able. But, I'm just saying that other formats/libraries support this natively out of the box.

[ Parent | on HN ]

Re:MessagePack: like JSON, but fast and small by thedance on Wednesday March 11, 2020 @12:59PM (#22547410)

It sounds a bit like you invented RecordIO[1]. I didn't realize you meant decoding from a stream containing several protobufs. In that case you do indeed need some kind of framing, because there is no end-of-message delimiter. The positive tradeoff is you can easily concatenate several protobufs into another, valid protobuf. That is impossible if there is a terminating symbol.

1: https://www.tensorflow.org/tutorials/load_data/tfrecord#tfre...

[ Parent | on HN ]

Re:MessagePack: like JSON, but fast and small by neuland on Wednesday March 11, 2020 @02:34PM (#22548793)

Oh neat. I'm not familiar with the ML/AI ecosystem. So, I didn't think to look there for formats; a bit of a shame I reinvented the wheel here, but it worked great for my specific use case.

Yeah, it's all about the tradeoffs with formats. Protobuf is a good choice a lot of the time and I always think about it as one of the first/preferred options. gRPC is great too.

[ Parent | on HN ]