Re:Zstandard – Real-time data compression algorithm
by pmarreck on Monday September 05, 2022 @12:23AM
(#32720477)
can't zstandard use standardized dictionaries as well?
[ Parent
| on HN ]
Re:Zstandard – Real-time data compression algorithm
by cyann on Monday September 05, 2022 @03:11AM
(#32721416)
Yes, sort of: "Zstd offers a training mode, which can be used to tune the algorithm for a selected type of data. Training Zstandard is achieved by providing it with a few samples (one file per sample). The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression. Using this dictionary, the compression ratio achievable on small data improves dramatically."
https://github.com/facebook/zstd#the-case-for-small-data-com...
Re:Zstandard – Real-time data compression algorithm
by PoignardAzur on Monday September 05, 2022 @03:49AM
(#32721616)
According to the Bugzilla thread, Facebook won't bother going through the process of publishing standardized dictionaries.
Re:Zstandard – Real-time data compression algorithm
by unsigner on Monday September 05, 2022 @04:15AM
(#32721755)
I wouldn't go to the trouble of determining the privacy implications of standartized dictionaries built from user data.
Re:Zstandard – Real-time data compression algorithm
by wongarsu on Monday September 05, 2022 @07:45AM
(#32723240)
But surely anyone could publish dictionaries? Like let's say Mozilla could publish a dictionary trained on HTTP headers, HTML and Javascript, and then accept traffic compressed with it under a "Accept-Encoding: zstd/mozilla" header