Re:AWS switch from gzip to zstd – about 30% reduction in com...
by DedlySnek on Sunday August 21, 2022 @04:13AM
(#32538294)
Yep, someone from aws clarified this on r/programming:
https://old.reddit.com/r/programming/comments/wtd61q/aws_swi...
He's talking about runtime logs of the S3 service, or possibly logs from Amazon's services that are written to S3.
Amazon has ~100K services running on millions of EC2 instances. (Generalizing) each service writes logs to the local machine, and then later compresses and transmits them to a set of systems that are backed by S3 for monitoring / further bulk processing etc.
Some back of the envelope math on this:
For 1 exabyte to be the number, that equates to < 1TB average logs kept per host. Depending on what the service is doing log retention is 3mo / 1y / 10y. Using those numbers we're talking per service average 12-460 MB/hour.