News for hackers. Stuff that matters.

A comment on We compress Pub/Sub messages and more, saving a load of money.

Re:We compress Pub/Sub messages and more, saving a load of m... by dilatedmind on Sunday January 03, 2021 @01:01PM (#25623509)
did you use any kind of message framing? Got bit by this at a previous job where we needed to change the message format to improve compression. Wound up figuring something out, but would have been easier if we had reserved a byte for versioning.
[ Parent | on HN ]
Re:We compress Pub/Sub messages and more, saving a load of m... by lawrjone on Sunday January 03, 2021 @04:55PM (#25625615)
I'm not quite sure what you mean by message framing.

If you mean marking messages as having been compressed, then absolutely yet. The Pub/Sub messages were tagged with `compress=true|false` so we could update downstream and upstream independently.

If you mean buffering log messages into a batched 'frame', then yes we did do this. We were taking about 500 log entries and compressing them into a single message, which was part of why the compression was so effective.

If you mean something different, then I'm at a loss and very interested in what you meant by the term!

[ Parent | on HN ]