I believe zstd can detect incompressible data quickly and falls back to a low-overhead passthrough mode in that case. So you can just run everything through zstd and it handles those cases pretty optimally already.

Hmm this made me chuckle, is that not equivalent to a test for randomness, which is very very hard indeed?

It is not. Just look at the frequencies in the data (histogram) and compute the order 0 entropy. Then you know if your data is compressible.

I think a useful test for this case could be pretty simple. Compress maybe 10% of the file (up to some reasonable maximum, like maybe a few megabytes) and check the compression ratio. If it's not something better than maybe 1.1x, give up, throw out the compressed data, and just pass through the file. Otherwise keep going.

It's written through a virtual file system which stores in blocks and not files, the implementation of that could happily run zstd on the way to the multiple physical disks.

This isn't about them shrinking your individual files, its compression on their internal transport.

If the customers stored .gz/.zip/... files you could transparently transcompress them to .zstd and back (with an added test of course)

It seems like that would cause problems. Right off the bat, any integrity hashes like md5 or sha256 for the original compressed files would likely be corrupted. Also, the compressed archive could have been structurally baked in a specific way that's meaningful to the customer. Zip archives in particular can have arbitrary data pretended to them.

I suppose you could speculatively decompress and then re-compress and see if you get the original compressed file back, and maybe most people happen to use the same compression implementations with default settings.

> Also, the compressed archive could have been structurally baked in a specific way that's meaningful to the customer.

EPUB is an example of this. They're mostly bog-standard ZIP archives, but in order for the file to be valid, the "first" file in the archive, linearly speaking, must be named "mimetype" and stored with no compression (each file in a ZIP archive can have a different compression level). If you just unzip an EPUB and then just dumbly zip it back up again, your end file will not be a valid EPUB.

Besides ZIP metadata, there may be flushes in specific points in the deflate stream (which applies to .gz files as well). These flushes reset the compression dictionary and make further compressed data independent from previous data (at the expense of losing some compression efficiency). So: AWS S3 customer may have injected these flushes to their .gz files (for whatever reason, e.g. steganography), and after gzip-to-zstd-to-gzip transcompression this steganographic data will be be lost (and of course sha256 and other similar hashes will be different, as you already said).

Exactly. I've built several logging systems over the years that intentionally flush compression streams (or concatenate gz streams) for robustness reasons.

You would only apply the compression on the internally stored file and then decompress when retrieving it for the customer again. That way all the hashes and original structure of the user are retained.

You would need to be able to reconstruct the input file bit-by-bit. S3 Clients expect to get back what they sent, exactly.

This puts a serious limitation on your compression. You would only be able to re-do the entropy coding part of DEFLATE, which is actually pretty good.

You would still need to store the original Huffman tables for each block, so you can reconstruct the entropy coding exactly.

I doubt this would even gain you a single percentage.

That wastes a lot of CPU compared to just running it through zstd.

There are tools like preflate [1] or precomp [2] that guarantees a bitwise identical reconstruction, of course modulo bugs.

[1] https://github.com/deus-libri/preflate

[2] https://github.com/schnaader/precomp-cpp/ (which internally makes use of preflate)

Certainly, if used as a data store for a data-lake-like solution, it's much more convenient to store data in uncompressed form, knowing that the underlying storage solution will take care of that for you. This way, you can just dump Json files in without it feeling like a collosal waste (or having to do extra work of converting them to a compressed format, such as Avro).

Storage level compression usually benefits whomever owns the storage.

I doubt Amazon intends to bill on compressed size.