News for hackers. Stuff that matters.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by sigmoid10 on Saturday August 20, 2022 @07:15AM (#32531119)
I think a lot if datacenter SSDs already come with in-drive hardware compression these days, since it not only increases speed but also longevity. So it would actually save money anyways.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by snoopy_telex on Saturday August 20, 2022 @10:41AM (#32532378)
They do not. It would be difficult to plan correctly if your free disk space is… variable.

Example:

You have an existing 40 gigabyte file

It happened to compress well

You delete it and your free disk space goes up by 4 gigabytes.

You then write a new 40 gigabyte file that doesn’t compress well

Replacing an existing file of the same size just ate an extra 36 gigabytes.

How would you plan around that? SSDs should store the bytes given and don’t play fancy games.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by sp332 on Saturday August 20, 2022 @11:03AM (#32532526)
It's not great, but it's also not unheard of. Tape capacities are often quoted at double the actual storage space, with fine print that says "assuming 50% compression". Also, if compression makes IOPs faster or reduces wear on SSDs, people might not complain so much.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by ChrisLomont on Saturday August 20, 2022 @11:02AM (#32532521)
SSDs could still do it for speed and to write less pages, making the drive last longer, and simply report the uncompressed space as used. They already do all sorts of tricks on pages such as moving them logically, having more internally than they report to use as pages wear out, and so on.

Given that, it's good sense to compress if at all possible simply to make the drive live longer.

And guess what - I just googled, tons of hits, and this has been done for a long time :)

So it makes sense, is done, and is important for modern SSD behavior.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by tgsovlerkhgsel on Saturday August 20, 2022 @11:19AM (#32532646)
None of this works if the data is encrypted.

I'm surprised (and shocked) that letting unencrypted data hit the disk is still common enough to make such optimizations worth it.

Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by goodpoint on Saturday August 20, 2022 @02:44PM (#32534263)
> I'm surprised (and shocked) that letting unencrypted data hit the disk is still common enough

There's very, very little benefit in encrypting data at a filesystem level in a datacenter if you think about it.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by tgsovlerkhgsel on Thursday August 25, 2022 @10:47AM (#32594372)
I explained why I think it does provide a benefit even in a datacenter:

"Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal."

Would you resell usable drives that you no longer want to use (e.g. because they're too small or your needs changed more towards SSDs) if unencrypted data was written to them at some point?

How much effort would you put into making sure that no broken drive that can't be wiped leaves the datacenter without shredding?

Any process you put in place will have gaps - e.g. through human error or malicious acts - and this provides pretty solid protection against that.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by staticassertion on Saturday August 20, 2022 @06:06PM (#32535576)
Unfortunately that's going to really depend. For example, if your threat model is "hard drive gets stolen" there's no point. If your threat model is "attacker can access my database" encrypting the data at the DB level does make sense. But it obviously breaks compression.

And unfortunately compression and encryption are seemingly at odds fundamentally :c

Re:AWS switch from gzip to zstd – about 30% reduction in com... by ChrisLomont on Saturday August 20, 2022 @01:18PM (#32533647)
Plenty of SSDs implement strong encryption, and there's an entire standard API for it. Why waste cpu on tasks your hardware already does?

Bitlocker supports on drive hardware encryption, and I'd be surprised if other major file systems didn't.

If I recall, it's a FIPS requirement for data at rest now.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by gspr on Sunday August 21, 2022 @04:08AM (#32538270)
> Why waste cpu on tasks your hardware already does?

Because the storage hardware might be loyal to someone who is not the owner of the data.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by tgsovlerkhgsel on Thursday August 25, 2022 @10:41AM (#32594294)
> Why waste cpu on tasks your hardware already does?

Because the hardware cannot be trusted to do it correctly. IIRC Bitlocker stopped relying on it for this reason.

https://www.howtogeek.com/fyi/you-cant-trust-bitlocker-to-en... (see the updates)

Re:AWS switch from gzip to zstd – about 30% reduction in com... by sigmoid10 on Saturday August 20, 2022 @09:04PM (#32536512)
Yes they do. Intel writes this for example (not all drives use it, but some definitely do):

>Data compression via encoding algorithms enables a solid state drive (SSD) to write less data, which in turn yields higher write bandwidth. With a significant amount of data being compressible, performance benefits can be substantial.

https://www.intel.com/content/www/us/en/support/articles/000...

Re:AWS switch from gzip to zstd – about 30% reduction in com... by anamexis on Saturday August 20, 2022 @10:58AM (#32532491)
Re:AWS switch from gzip to zstd – about 30% reduction in com... by wmf on Saturday August 20, 2022 @01:12PM (#32533602)
Note that these are pre-2017 consumer SSDs. I think SSD compression fell out of favor due to the rise of FDE.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by ChrisLomont on Saturday August 20, 2022 @01:20PM (#32533665)
FDE can be done on the SSD after compression, and often is.

Adding life to SSDs is a terribly useful feature

Re:AWS switch from gzip to zstd – about 30% reduction in com... by natmaka on Saturday August 20, 2022 @11:25AM (#32532701)
Isn't it solved by adding an abstraction layer between the storage and the filesystem, the way "vdo" ( https://github.com/dm-vdo/vdo ) does it?
Re:AWS switch from gzip to zstd – about 30% reduction in com... by yunohn on Saturday August 20, 2022 @04:03PM (#32534783)
What do you mean? Even without compression, you have half the problem. Obviously, at scale, it’s all just statistics and planning on trends instead of individual files.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by collegeburner on Saturday August 20, 2022 @11:51AM (#32532921)
how is this any different than e.g. fs level compression with zstd on btrfs? seems enough people find that useful.