News for hackers. Stuff that matters.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by sexy_panda on Saturday August 20, 2022 @08:00AM (#32531362)
I wonder if this could be done on a FPGA..
Re:AWS switch from gzip to zstd – about 30% reduction in com... by vasco on Saturday August 20, 2022 @08:16AM (#32531457)
FPGA for development, AWS can afford to order custom ICs.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by toomuchtodo on Saturday August 20, 2022 @10:28PM (#32536887)
Could surplus crypto miner GPUs be repurposed for this?
Re:AWS switch from gzip to zstd – about 30% reduction in com... by vasco on Sunday August 21, 2022 @03:19AM (#32538056)
No
Re:AWS switch from gzip to zstd – about 30% reduction in com... by Frost1x on Saturday August 20, 2022 @08:10AM (#32531426)
With enough gates you can do anything on an FPGA
Re:AWS switch from gzip to zstd – about 30% reduction in com... by pclmulqdq on Saturday August 20, 2022 @09:37AM (#32531975)
How much would someone pay for this? I have a half-written zstd core, but I doubt the market for $100-150 FPGA-based compression accelerators is all that large.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by tgsovlerkhgsel on Saturday August 20, 2022 @11:21AM (#32532662)
Probably not worth it as a FPGA solution or even in general as an add-on card (the overhead of dealing with such extra hardware means that the threshold for "worth it" is very high).

I would expect this to become part of newer generations of CPUs once it becomes popular.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by wmf on Saturday August 20, 2022 @01:11PM (#32533594)
Intel sells a gzip PCIe card called QAT. Not many seem to be sold.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by NelsonMinar on Saturday August 20, 2022 @02:07PM (#32533997)
I'd never heard of this! The full name is QuickAssist and it does encryption too. They advertise 100Gb/s symmetric crypto, 70Gb/s compression (or roughly 100x faster than ZStandard on a single CPU). Seems to retail for about $650 for a card.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by sexy_panda on Sunday August 28, 2022 @10:28AM (#32628638)
It seems like QuickAssist can also be used for Transcoding. Plex e.g. supports it.

My Intel i9 11900k seems to support it or has it built-in.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by pclmulqdq on Saturday August 20, 2022 @02:14PM (#32534043)
QAT is actually likely to end up inside new server CPUs from Intel - at least according to the advertising material. Also, it is in their new SmartNICs. At least somebody is using it.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by zxcvbn4038 on Saturday August 20, 2022 @02:16PM (#32534058)
It has been done before, if you offload gzip you turn your PCI bus into a choke point. Most of the time you do better keeping it on the main processor.