News for hackers. Stuff that matters.
Re:The Elegance of Deflate by xenadu02 on Monday August 22, 2016 @03:35AM (#12334685)
The cost-benefit is just too low for this to be practical in most cases:

* The large data being transferred is almost never text; it's video, images, or audio which is already using advanced (sometimes lossy) compression * Even if it is text, it's probably in some document format or HTML which greatly complicates the shared dictionary scenario (and would require a separate dictionary for each format)

In reality even deflate is already doing what you propose, but it builds the dictionary dynamically and includes it with each message. The up side is its dictionary can find common phrases and patterns that wouldn't be in your fixed dictionary. As the size of the text becomes larger the dictionary becomes a smaller and smaller percentage of the total.

Re:The Elegance of Deflate by creshal on Monday August 22, 2016 @04:14AM (#12334817)
Even text is usually precompressed. Both OOXML and ODF use deflate internally, epub as well, and PDF has such a high bloat-to-text ratio that compressing only the text is moot (assuming you don't have one of the increasingly common PDFs where even the text is rendered as image, aaaaargh…).
Re:The Elegance of Deflate by Grishnakh on Monday August 22, 2016 @02:21PM (#12338270)
I've looked into PDFs a little for a personal project of mine that worked with auto-generated PDFs, and I've found they can really vary wildly in the amount of bloat, and I've also found that much of the bloat seems to be from including fonts. If you actually look at the PDF file's binary, it's not that hard to understand, and the actual text is stored mainly as ASCII/utf8 I think. So, if I wanted to really efficiently store a bunch of PDFs that come from the same source, it seems like it should be possible to copy the "bloat" sections (likely embedded fonts) which are all completely identical, and use those in a dictionary in a custom compressor, and then just use zlib for the rest.

I've also noticed that some PDF generators include far, far more bloat than others, though I'm not sure why.