AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage

Re:AWS switch from gzip to zstd – about 30% reduction in com... by wmf on Saturday August 20, 2022 @01:11PM (#32533594)

Intel sells a gzip PCIe card called QAT. Not many seem to be sold.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by NelsonMinar on Saturday August 20, 2022 @02:07PM (#32533997)

I'd never heard of this! The full name is QuickAssist and it does encryption too. They advertise 100Gb/s symmetric crypto, 70Gb/s compression (or roughly 100x faster than ZStandard on a single CPU). Seems to retail for about $650 for a card.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by sexy_panda on Sunday August 28, 2022 @10:28AM (#32628638)

It seems like QuickAssist can also be used for Transcoding. Plex e.g. supports it.

My Intel i9 11900k seems to support it or has it built-in.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by pclmulqdq on Saturday August 20, 2022 @02:14PM (#32534043)

QAT is actually likely to end up inside new server CPUs from Intel - at least according to the advertising material. Also, it is in their new SmartNICs. At least somebody is using it.

[ Parent | on HN ]

Re:AWS switch from gzip to zstd – about 30% reduction in com... by zxcvbn4038 on Saturday August 20, 2022 @02:16PM (#32534058)

It has been done before, if you offload gzip you turn your PCI bus into a choke point. Most of the time you do better keeping it on the main processor.

[ Parent | on HN ]