News for hackers. Stuff that matters.

A comment on MessagePack: like JSON, but fast and small. Parent comment

Re:MessagePack: like JSON, but fast and small by thedance on Wednesday March 11, 2020 @12:59PM (#22547410)
It sounds a bit like you invented RecordIO[1]. I didn't realize you meant decoding from a stream containing several protobufs. In that case you do indeed need some kind of framing, because there is no end-of-message delimiter. The positive tradeoff is you can easily concatenate several protobufs into another, valid protobuf. That is impossible if there is a terminating symbol.

1: https://www.tensorflow.org/tutorials/load_data/tfrecord#tfre...

[ Parent | on HN ]
Re:MessagePack: like JSON, but fast and small by neuland on Wednesday March 11, 2020 @02:34PM (#22548793)
Oh neat. I'm not familiar with the ML/AI ecosystem. So, I didn't think to look there for formats; a bit of a shame I reinvented the wheel here, but it worked great for my specific use case.

Yeah, it's all about the tradeoffs with formats. Protobuf is a good choice a lot of the time and I always think about it as one of the first/preferred options. gRPC is great too.

[ Parent | on HN ]