News for hackers. Stuff that matters.
Re:Drink Me: (Ab)Using a LLM to Compress Text by Intralexical on Friday May 03, 2024 @11:21PM (#40254542)
IDK why, but BZIP2 seems to do somewhat better that other compression algorithms for natural language text:

    $ curl https://www.gutenberg.org/cache/epub/11/pg11.txt | bzip2 --best | wc
        246    1183   48925
Also, ZSTD goes all the way up to `--ultra -22` plus `--long=31` (4GB window— Irrelevant here since the file fits in the default 8MB anyway).
Re:Drink Me: (Ab)Using a LLM to Compress Text by pastage on Sunday May 05, 2024 @04:42AM (#40263204)
You can use a preshared dictionary with bzip2 and zstd so you can get that down alot by using different dictionaries depending on certain rules. I dont know if it helps with literature but I had great success in sending databases with free text like that. In the end it was easier to just use one dictionary for everything and just skip the rules.