We compress Pub/Sub messages and more, saving a load of money

Re:We compress Pub/Sub messages and more, saving a load of m... by saurik on Sunday January 03, 2021 @02:08PM (#25624141)

Data sent to S3 is usually hashed (depending on authentication type) in addition to being transport encrypted; I imagine the majority of this cost here is the encryption of a larger payload (which many would consider indispensable, but I point this out because I do not generally assume this when I merely consider "over the network").

[ Parent | on HN ]

Re:We compress Pub/Sub messages and more, saving a load of m... by user5994461 on Sunday January 03, 2021 @02:30PM (#25624387)

You assume incorrectly. SSL encryption is in the order of 1 GB/s on a recent CPU with AES instructions (anything from this decade).

Gzip is in the order of 10 MB/s with default settings, down to 1 MB/s with the strongest compression setting. It's really really slow.

[ Parent | on HN ]

Re:We compress Pub/Sub messages and more, saving a load of m... by Kranar on Sunday January 03, 2021 @03:28PM (#25624928)

GNU gzip the application is slow on the order of 10 MB/s because of how it does file IO, but the DEFLATE algorithm that gzip is based off of is much faster than 10 MB/s at the default "level 6". For example the slz implementation of DEFLATE compresses text at 1 GB/s [1]. Even the fairly common zlib implementation can compress text at close to 300 MB/s.

http://www.libslz.org/

[ Parent | on HN ]

Re:We compress Pub/Sub messages and more, saving a load of m... by user5994461 on Sunday January 10, 2021 @09:01AM (#25714678)

DEFLATE at level 6 is really doing 10 MB/s, doesn't matter if you're using gzip or zlib or another library.

slz is closer to level 20 (if there was a level 20). It's fast but the compression ratio is meh. You're better of using lz4 or zstd.

[ Parent | on HN ]