MessagePack: like JSON, but fast and small

Re:MessagePack: like JSON, but fast and small by thedance on Tuesday March 10, 2020 @02:03PM (#22537851)

There's nothing fundamental about protobuf itself that prevents streaming output. You just have to make a pass over the proto structure first to compute the sizes, then stream it out using the precomputed sizes. At no time do you necessarily need the entire representation of the output in memory. The C++ library, for example, offers this.

[ Parent | on HN ]

Re:MessagePack: like JSON, but fast and small by neuland on Tuesday March 10, 2020 @02:39PM (#22538196)

Yes, and that's what I did. In my case, I was writing to a file. And my languages were C, Python, and Lua. So, I wrote a small package (plus bindings for Python and Lua) that would write out the message length as a uint64, then a uint8 for which schema the message is (I was working with multiple different message types), and then the actual message bytes.

So, it's do-able. But, I'm just saying that other formats/libraries support this natively out of the box.

[ Parent | on HN ]

Re:MessagePack: like JSON, but fast and small by thedance on Wednesday March 11, 2020 @12:59PM (#22547410)

It sounds a bit like you invented RecordIO[1]. I didn't realize you meant decoding from a stream containing several protobufs. In that case you do indeed need some kind of framing, because there is no end-of-message delimiter. The positive tradeoff is you can easily concatenate several protobufs into another, valid protobuf. That is impossible if there is a terminating symbol.

1: https://www.tensorflow.org/tutorials/load_data/tfrecord#tfre...

[ Parent | on HN ]

Re:MessagePack: like JSON, but fast and small by neuland on Wednesday March 11, 2020 @02:34PM (#22548793)

Oh neat. I'm not familiar with the ML/AI ecosystem. So, I didn't think to look there for formats; a bit of a shame I reinvented the wheel here, but it worked great for my specific use case.

Yeah, it's all about the tradeoffs with formats. Protobuf is a good choice a lot of the time and I always think about it as one of the first/preferred options. gRPC is great too.

[ Parent | on HN ]