Hardware compression IS available in Graviton 2: "1Tbit/s of compression accelerators • 2xlarge and larger instances will have a compression device • DPDK and Linux kernel drivers will be available ahead of GA • Data compression at up to 15GB/s and decompression at up to 11GB/"

I wonder if this could be done on a FPGA..

FPGA for development, AWS can afford to order custom ICs.

Could surplus crypto miner GPUs be repurposed for this?

No

With enough gates you can do anything on an FPGA

How much would someone pay for this? I have a half-written zstd core, but I doubt the market for $100-150 FPGA-based compression accelerators is all that large.

Probably not worth it as a FPGA solution or even in general as an add-on card (the overhead of dealing with such extra hardware means that the threshold for "worth it" is very high).

I would expect this to become part of newer generations of CPUs once it becomes popular.

Intel sells a gzip PCIe card called QAT. Not many seem to be sold.

I'd never heard of this! The full name is QuickAssist and it does encryption too. They advertise 100Gb/s symmetric crypto, 70Gb/s compression (or roughly 100x faster than ZStandard on a single CPU). Seems to retail for about $650 for a card.

It seems like QuickAssist can also be used for Transcoding. Plex e.g. supports it.

My Intel i9 11900k seems to support it or has it built-in.

QAT is actually likely to end up inside new server CPUs from Intel - at least according to the advertising material. Also, it is in their new SmartNICs. At least somebody is using it.

It has been done before, if you offload gzip you turn your PCI bus into a choke point. Most of the time you do better keeping it on the main processor.

I think a lot if datacenter SSDs already come with in-drive hardware compression these days, since it not only increases speed but also longevity. So it would actually save money anyways.

They do not. It would be difficult to plan correctly if your free disk space is… variable.

Example:

You have an existing 40 gigabyte file

It happened to compress well

You delete it and your free disk space goes up by 4 gigabytes.

You then write a new 40 gigabyte file that doesn’t compress well

Replacing an existing file of the same size just ate an extra 36 gigabytes.

How would you plan around that? SSDs should store the bytes given and don’t play fancy games.

It's not great, but it's also not unheard of. Tape capacities are often quoted at double the actual storage space, with fine print that says "assuming 50% compression". Also, if compression makes IOPs faster or reduces wear on SSDs, people might not complain so much.

SSDs could still do it for speed and to write less pages, making the drive last longer, and simply report the uncompressed space as used. They already do all sorts of tricks on pages such as moving them logically, having more internally than they report to use as pages wear out, and so on.

Given that, it's good sense to compress if at all possible simply to make the drive live longer.

And guess what - I just googled, tons of hits, and this has been done for a long time :)

So it makes sense, is done, and is important for modern SSD behavior.

None of this works if the data is encrypted.

I'm surprised (and shocked) that letting unencrypted data hit the disk is still common enough to make such optimizations worth it.

Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal.

> I'm surprised (and shocked) that letting unencrypted data hit the disk is still common enough

There's very, very little benefit in encrypting data at a filesystem level in a datacenter if you think about it.

I explained why I think it does provide a benefit even in a datacenter:

"Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal."

Would you resell usable drives that you no longer want to use (e.g. because they're too small or your needs changed more towards SSDs) if unencrypted data was written to them at some point?

How much effort would you put into making sure that no broken drive that can't be wiped leaves the datacenter without shredding?

Any process you put in place will have gaps - e.g. through human error or malicious acts - and this provides pretty solid protection against that.

Unfortunately that's going to really depend. For example, if your threat model is "hard drive gets stolen" there's no point. If your threat model is "attacker can access my database" encrypting the data at the DB level does make sense. But it obviously breaks compression.

And unfortunately compression and encryption are seemingly at odds fundamentally :c

Plenty of SSDs implement strong encryption, and there's an entire standard API for it. Why waste cpu on tasks your hardware already does?

Bitlocker supports on drive hardware encryption, and I'd be surprised if other major file systems didn't.

If I recall, it's a FIPS requirement for data at rest now.

> Why waste cpu on tasks your hardware already does?

Because the storage hardware might be loyal to someone who is not the owner of the data.

> Why waste cpu on tasks your hardware already does?

Because the hardware cannot be trusted to do it correctly. IIRC Bitlocker stopped relying on it for this reason.

https://www.howtogeek.com/fyi/you-cant-trust-bitlocker-to-en... (see the updates)

Yes they do. Intel writes this for example (not all drives use it, but some definitely do):

>Data compression via encoding algorithms enables a solid state drive (SSD) to write less data, which in turn yields higher write bandwidth. With a significant amount of data being compressible, performance benefits can be substantial.

https://www.intel.com/content/www/us/en/support/articles/000...

Yes they do.

https://www.intel.com/content/www/us/en/support/articles/000...

Note that these are pre-2017 consumer SSDs. I think SSD compression fell out of favor due to the rise of FDE.

FDE can be done on the SSD after compression, and often is.

Adding life to SSDs is a terribly useful feature

Isn't it solved by adding an abstraction layer between the storage and the filesystem, the way "vdo" ( https://github.com/dm-vdo/vdo ) does it?

What do you mean? Even without compression, you have half the problem. Obviously, at scale, it’s all just statistics and planning on trends instead of individual files.

how is this any different than e.g. fs level compression with zstd on btrfs? seems enough people find that useful.