We compress Pub/Sub messages and more, saving a load of money

Re:We compress Pub/Sub messages and more, saving a load of m... by Kranar on Sunday January 03, 2021 @03:28PM (#25624928)

GNU gzip the application is slow on the order of 10 MB/s because of how it does file IO, but the DEFLATE algorithm that gzip is based off of is much faster than 10 MB/s at the default "level 6". For example the slz implementation of DEFLATE compresses text at 1 GB/s [1]. Even the fairly common zlib implementation can compress text at close to 300 MB/s.

http://www.libslz.org/

[ Parent | on HN ]

Re:We compress Pub/Sub messages and more, saving a load of m... by user5994461 on Sunday January 10, 2021 @09:01AM (#25714678)

DEFLATE at level 6 is really doing 10 MB/s, doesn't matter if you're using gzip or zlib or another library.

slz is closer to level 20 (if there was a level 20). It's fast but the compression ratio is meh. You're better of using lz4 or zstd.

[ Parent | on HN ]