Re:We compress Pub/Sub messages and more, saving a load of m...
by lawrjone on Sunday January 03, 2021 @04:55PM
(#25625615)
I'm not quite sure what you mean by message framing.
If you mean marking messages as having been compressed, then absolutely yet. The Pub/Sub messages were tagged with `compress=true|false` so we could update downstream and upstream independently.
If you mean buffering log messages into a batched 'frame', then yes we did do this. We were taking about 500 log entries and compressing them into a single message, which was part of why the compression was so effective.
If you mean something different, then I'm at a loss and very interested in what you meant by the term!
If you mean marking messages as having been compressed, then absolutely yet. The Pub/Sub messages were tagged with `compress=true|false` so we could update downstream and upstream independently.
If you mean buffering log messages into a batched 'frame', then yes we did do this. We were taking about 500 log entries and compressing them into a single message, which was part of why the compression was so effective.
If you mean something different, then I'm at a loss and very interested in what you meant by the term!