FTA: "This post aimed to cover a scenario where the cost of compression, in both compute resource and time-to-build, was significantly outweighed by the savings it would make."

Author here! A few others have pointed this out, but to restate: in our situation, compression was costing us almost nothing.

60TB of logs was 60 days worth of retention, so 1TB a day. That means we process about 11MB/s on average, peaking at 100MB/s.

A single CPU core can manage 100MB/s of compression, so if you assume we compress and decompress in multiple places, let's say we're paying about 4 CPU cores constantly for this.

That's a pretty worse case scenario, and it would cost us $37.50/month on GCP for those cores, in order to save about 100x that amount.

The takeaway (for me at least) is that compressing an in-flight transfer is almost always worthwhile if you're working in the cloud and the transfer will eventually be stored somewhere. The economics of CPU vs the total amount of data storage cost is a no brainer.

Hope that makes sense!

Thanks for the clarification.

my intuition is you can save cpu time when compressing before sending over the network (and certainly wall time).

a quick test copying a 24M file (with similiar compression ratios) to s3 showed a 6% decrease in cpu time when piping through gzip.

Depends on selected compressor, but yes, you can. I've definitely observed zstd-1 to be a net savings, where compression/decompression costs were offset by pushing fewer bytes through the RPC and network layers - and this was only from observing the endpoints, not even counting CPU on intermediate proxies/firewalls/etc.

I wouldn't normally expect gzip to be a net savings (it's comparatively more expensive), but depending on compression ratio achieved and what layers you're passing the bytes through, I'd definitely believe it can be in some contexts.

Data sent to S3 is usually hashed (depending on authentication type) in addition to being transport encrypted; I imagine the majority of this cost here is the encryption of a larger payload (which many would consider indispensable, but I point this out because I do not generally assume this when I merely consider "over the network").

You assume incorrectly. SSL encryption is in the order of 1 GB/s on a recent CPU with AES instructions (anything from this decade).

Gzip is in the order of 10 MB/s with default settings, down to 1 MB/s with the strongest compression setting. It's really really slow.

GNU gzip the application is slow on the order of 10 MB/s because of how it does file IO, but the DEFLATE algorithm that gzip is based off of is much faster than 10 MB/s at the default "level 6". For example the slz implementation of DEFLATE compresses text at 1 GB/s [1]. Even the fairly common zlib implementation can compress text at close to 300 MB/s.

http://www.libslz.org/

DEFLATE at level 6 is really doing 10 MB/s, doesn't matter if you're using gzip or zlib or another library.

slz is closer to level 20 (if there was a level 20). It's fast but the compression ratio is meh. You're better of using lz4 or zstd.