Re:Drink Me: (Ab)Using a LLM to Compress Text
by Intralexical on Friday May 03, 2024 @11:21PM
(#40254542)
IDK why, but BZIP2 seems to do somewhat better that other compression algorithms for natural language text:
Also, ZSTD goes all the way up to `--ultra -22` plus `--long=31` (4GB window— Irrelevant here since the file fits in the default 8MB anyway).
[ Parent
| on HN ]
Re:Drink Me: (Ab)Using a LLM to Compress Text
by pastage on Sunday May 05, 2024 @04:42AM
(#40263204)
You can use a preshared dictionary with bzip2 and zstd so you can get that down alot by using different dictionaries depending on certain rules. I dont know if it helps with literature but I had great success in sending databases with free text like that. In the end it was easier to just use one dictionary for everything and just skip the rules.