News for hackers. Stuff that matters.
Re:Improving compression with a preset DEFLATE dictionary by marcosdumay on Monday March 30, 2015 @11:30AM (#9290145)
> You could use variable length indexing method to minimise that third point though, much like UTF8 for text - the vast majority of references need not require close to the 40 bits.

It's the other way around. If you used variable length indexes, most of your symbols will be much larger than 40 bits.

Take as an example a set of 256 byte long symbols. If you shorten one of them to a "1" bit, you just used half of the symbols you could represent with that byte, and almost half of them will need to grow at least one bit.

Re:Improving compression with a preset DEFLATE dictionary by dspillett on Tuesday March 31, 2015 @05:45AM (#9295200)
Many symbols being longer only matters if they are common (and your encoding method doesn't allow you to leave affected sequences plain to avoid increasing their size), and if, moving away from UFT8-a-like encodings which might have been a bad example, you allow context beyond one symbol you can break them into pages making symbols after a page selection smaller (though the next one that requires a new page to be selected will necessarily be larger).

There are various ways you could arrange the symbol encoding and how deal with those parts of the input that can't be encoded effectively (where the encoding would increase size overall instead of decreasing it). I'm pretty sure it would work, though whether it would be truly practical (in terms of the return from making the effort) rather than just being an interesting little thought experiment is definitely a valid discussion point! The idea of a collection of preset initial dictionaries for LZ77 and similar feels much more practical than the single large reference dictionary for some other scheme.