Re:Improving compression at scale with Zstandard
by JyrkiAlakuijala on Wednesday December 19, 2018 @06:36PM
(#18721053)
Brotli added custom dictionaries in 2014 or 2015 (before zstd format was even frozen). We are currently working on a new more efficient custom dictionary that uses context and transforms to ramp up the impact of the custom dictionary (by about 20%).
Re:Improving compression at scale with Zstandard
by pmarreck on Wednesday December 19, 2018 @05:40PM
(#18720554)
Browsers could cache dictionaries per MIME type per domain which are provided by the server. Would be super cool, if a bit more complicated.
Re:Improving compression at scale with Zstandard
by felixhandte on Wednesday December 19, 2018 @06:03PM
(#18720776)
Yep, that's in the vicinity of the solutions we're thinking about. There are a few proposals out there, and prior art like SDCH [1].
The hard part is that compression is already an attack vector for the web (e.g., CRIME [2], BREACH [3], et al.). We want to make sure that we're not eroding or unduly complicating that situation [4].
[1] https://tools.ietf.org/html/draft-lee-sdch-spec-00 [2] https://en.wikipedia.org/wiki/CRIME [3] https://en.wikipedia.org/wiki/BREACH [4] https://tools.ietf.org/html/draft-kucherawy-httpbis-dict-sec...
I am working on that very question! Zstd's support for creating and using custom dictionaries opens the door to significant efficiencies. As described in the post though, dictionaries make compression a more complicated thing to use, and so there are lots of questions about how to apply that to the public internet in a way that's cross-compatible and secure. In short: it's something we're actively exploring.
[1] http://quixdb.github.io/squash-benchmark/unstable/?dataset=e...