News for hackers. Stuff that matters.
Re:Brotli compression for the web by amelius on Saturday December 26, 2015 @06:53AM (#10793925)
I'm hoping for a compression algorithm which allows for separated codebooks, so e.g. one can compactly send updates when previous versions have already been transferred.
Re:Brotli compression for the web by twotwotwo on Saturday December 26, 2015 @07:28PM (#10795850)
Chrome has something like this called SDCH, but it hasn't seen a ton of adoption outside Google:

https://en.wikipedia.org/wiki/SDCH

https://engineering.linkedin.com/shared-dictionary-compressi...

https://aprescott.com/posts/sdch

Guessing part of the hurdle is that you can't simply drop it in like gzip; you have to generate a dictionary and serve it and refresh it now and then (the LinkedIn post explains). And a CDN can't just cache "the SDCH'd version" of a resource; ideally it'd cache one version for each recently used dict.

The other, slightly happier "problem," is that other tuning can make the additional gains from SDCH less dramatic. I looked, and the Google homepage has a blob of JS that's 40KB gzip+sdch'd--but it's ~100KB gzip -9'd, and probably costs me 0KB most of the time because it's cacheable and its Last-Modified is about a month ago.

CloudFlare's Railgun proxy does use delta coding to good effect, though in a much different context (origin server to edge location).

I can imagine other variations, like priming Brotli's (de)compressor's history buffer with the first bytes of the old version of a cacheable resource in an If-None-Match request, if both sides have the old version handy and advertise the encoding. Sounds like what you're saying. Like SDCH, that still isn't drop-in; the server has to remember old versions of things. And it only applies to cached resources.

Anyhow, interesting space.

Re:Brotli compression for the web by twotwotwo on Sunday December 27, 2015 @04:22PM (#10798802)
> priming Brotli's (de)compressor's history buffer with...the old version...if both sides...advertise the encoding

Too late to edit, but I mean a separate Accept-Encoding value for the diff-like usage (brdiff or whatever), not piggy-backing on the value already being added for Brotli. Clearly if you're messing with the history buffer it's no longer Brotli-as-we-know-it.

Re:Brotli compression for the web by robryk on Saturday December 26, 2015 @06:59AM (#10793936)
What do you mean by a codebook? Something like a static dictionary?
Re:Brotli compression for the web by PeCaN on Saturday December 26, 2015 @01:26PM (#10794757)
For LZ algorithms, the dictionary made when compressing the first version would be reused for subsequent "versions" of the same document. So sort of a static dictionary, but specialized for what you're sending.

That or maybe he means the Huffman codes, in which case you'd do the same thing but s/LZ/Huffman/ s/dictionary/tree/g.

I tend to favor algorithms like LZF and LZ4 that don't have a separate dictionary for compressing text though, since there's just about no performance loss on either end and compression is "good enough".

Re:Brotli compression for the web by robryk on Monday December 28, 2015 @06:30PM (#10804130)
> For LZ algorithms, the dictionary made when compressing the first version would be reused for subsequent "versions" of the same document. So sort of a static dictionary, but specialized for what you're sending.

Ah, so you mean something like what SDCH does. This works well for files that are smaller than the window size and has rewards rapidly diminishing with filesize past that.