AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage

Re:AWS switch from gzip to zstd – about 30% reduction in com... by bick_nyers on Saturday August 20, 2022 @06:42AM (#32530951)

I'm surprised they aren't using an FPGA solution

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by hbbio on Saturday August 20, 2022 @08:17AM (#32531463)

Knowing such implementation exists, is open source, and since they don't state what they're using... you may just be right

https://pure.tudelft.nl/ws/portalfiles/portal/94907930/Jiany...

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by pclmulqdq on Saturday August 20, 2022 @10:18AM (#32532215)

FPGAs are really expensive in volume compared to other options, so it's not surprising to me.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by londons_explore on Saturday August 20, 2022 @06:20PM (#32535655)

But on an FPGA you can probably implement super fast and effective compression.

If it is far enough above the curve of speed Vs ratio, then it will pay for itself in saved storage.

Remember there is no need for said FPGA to be in every machine - in a data center with ten's of gigabits of bandwidth to every node, you can send data to another machine for compression and receive back the compressed data to store.

[ Parent | on HN ]