Re:AWS switch from gzip to zstd – about 30% reduction in com...
by sexy_panda on Saturday August 20, 2022 @08:00AM
(#32531362)
I wonder if this could be done on a FPGA..
[ Parent
| on HN ]
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by vasco on Saturday August 20, 2022 @08:16AM
(#32531457)
FPGA for development, AWS can afford to order custom ICs.
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by toomuchtodo on Saturday August 20, 2022 @10:28PM
(#32536887)
Could surplus crypto miner GPUs be repurposed for this?
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by vasco on Sunday August 21, 2022 @03:19AM
(#32538056)
No
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by Frost1x on Saturday August 20, 2022 @08:10AM
(#32531426)
With enough gates you can do anything on an FPGA
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by pclmulqdq on Saturday August 20, 2022 @09:37AM
(#32531975)
How much would someone pay for this? I have a half-written zstd core, but I doubt the market for $100-150 FPGA-based compression accelerators is all that large.
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by tgsovlerkhgsel on Saturday August 20, 2022 @11:21AM
(#32532662)
Probably not worth it as a FPGA solution or even in general as an add-on card (the overhead of dealing with such extra hardware means that the threshold for "worth it" is very high).
I would expect this to become part of newer generations of CPUs once it becomes popular.
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by wmf on Saturday August 20, 2022 @01:11PM
(#32533594)
Intel sells a gzip PCIe card called QAT. Not many seem to be sold.
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by NelsonMinar on Saturday August 20, 2022 @02:07PM
(#32533997)
I'd never heard of this! The full name is QuickAssist and it does encryption too. They advertise 100Gb/s symmetric crypto, 70Gb/s compression (or roughly 100x faster than ZStandard on a single CPU). Seems to retail for about $650 for a card.
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by sexy_panda on Sunday August 28, 2022 @10:28AM
(#32628638)
It seems like QuickAssist can also be used for Transcoding. Plex e.g. supports it.
My Intel i9 11900k seems to support it or has it built-in.
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by pclmulqdq on Saturday August 20, 2022 @02:14PM
(#32534043)
QAT is actually likely to end up inside new server CPUs from Intel - at least according to the advertising material. Also, it is in their new SmartNICs. At least somebody is using it.
Re:AWS switch from gzip to zstd – about 30% reduction in com...
by zxcvbn4038 on Saturday August 20, 2022 @02:16PM
(#32534058)
It has been done before, if you offload gzip you turn your PCI bus into a choke point. Most of the time you do better keeping it on the main processor.