News for hackers. Stuff that matters.
Re:Drink Me: (Ab)Using a LLM to Compress Text by pronoiac on Friday May 03, 2024 @04:47AM (#40245530)
I expected general-purpose compressors to do better, but I was wrong. For the whole document:

  174355 pg11.txt
   60907 pg11.txt.gz-9
   58590 pg11.txt.zstd-9
   54164 pg11.txt.xz-9
   25360 [from blog post]
Re:Drink Me: (Ab)Using a LLM to Compress Text by EgoIncarnate on Friday May 03, 2024 @10:59PM (#40254465)
Which model did you use?
Re:Drink Me: (Ab)Using a LLM to Compress Text by Intralexical on Friday May 03, 2024 @11:21PM (#40254542)
IDK why, but BZIP2 seems to do somewhat better that other compression algorithms for natural language text:

    $ curl https://www.gutenberg.org/cache/epub/11/pg11.txt | bzip2 --best | wc
        246    1183   48925
Also, ZSTD goes all the way up to `--ultra -22` plus `--long=31` (4GB window— Irrelevant here since the file fits in the default 8MB anyway).
Re:Drink Me: (Ab)Using a LLM to Compress Text by pastage on Sunday May 05, 2024 @04:42AM (#40263204)
You can use a preshared dictionary with bzip2 and zstd so you can get that down alot by using different dictionaries depending on certain rules. I dont know if it helps with literature but I had great success in sending databases with free text like that. In the end it was easier to just use one dictionary for everything and just skip the rules.
Re:Drink Me: (Ab)Using a LLM to Compress Text by immibis on Saturday May 04, 2024 @10:30PM (#40261848)
And here's the opposite - using gzip as a (merely adequate) "large" language model: https://aclanthology.org/2023.findings-acl.426.pdf

By the way, if you want to see how well gzip actually models language, take any gzipped file, flip a few bits, and unzip it. If it gives you a checksum error, ignore that. You might have to unzip in a streaming way so that it can't tell the checksum is wrong until it's already printed the wrong data that you want to see.

Re:Drink Me: (Ab)Using a LLM to Compress Text by anonu on Saturday May 04, 2024 @11:19PM (#40262017)
Shouldn't you factor in the size of the compression tools needed?
Re:Drink Me: (Ab)Using a LLM to Compress Text by TeMPOraL on Sunday May 05, 2024 @03:37AM (#40263014)
Not if you're interested in compressing more than one file - compression tools are a fixed, one-time cost in size.
Re:Drink Me: (Ab)Using a LLM to Compress Text by sp332 on Sunday May 05, 2024 @08:49AM (#40264475)
For the prize, yes. But it’s interesting to see that even a much larger decompressor might not necessarily buy you a smaller file.
Re:Drink Me: (Ab)Using a LLM to Compress Text by Legend2440 on Sunday May 05, 2024 @01:10AM (#40262417)
LLMs blow away traditional compressors because they are very good predictors, and prediction and compression are the same operation.

You can convert any predictor into a lossless compressor by feeding the output probabilities into an entropy coding algorithm. LLMs can get compression ratios as high as 95% (0.4 bits per character) on english text.

https://arxiv.org/html/2404.09937v1

Re:Drink Me: (Ab)Using a LLM to Compress Text by Der_Einzige on Sunday May 05, 2024 @01:41AM (#40262540)
The 1-1 correspondence between prediction and compression is one of the most counterintuitive and fascinating things in all of AI to me.
Re:Drink Me: (Ab)Using a LLM to Compress Text by mjburgess on Sunday May 05, 2024 @07:45AM (#40264111)
Its only equivalent for a very narrow sense of 'prediction' , namely modelling conditional probability distributions over known data.

There's no sense, for example, in which deriving a prediction about the nature of reality from a novel scientific theory is 'compression'

eg., suppose we didn't know a planet existed, and we looked at orbital data. There's no sense in which compressing that data would indicate another planet existed.

It's a great source of confusion that people think AI/ML systems are 'predicting' novel distributions of observations (science), vs., novel observations of the same distribution (statistics).

It should be more obvious that the latter is just compression, since it's just taking a known distribution of data and replacing it with a derivative optimal value.

Science predicts novel distributions based on theories, ie., it says the world is other than we previously supposed.

Re:Drink Me: (Ab)Using a LLM to Compress Text by palmtree3000 on Sunday May 05, 2024 @08:16AM (#40264278)
Sure it is! If we were trying to compress an archive of orbital data, one way to do it would be "initial positions + periodic error correction". If you have the new planet, your errors will be smaller and can be represented in less space at the same precision.
Re:Drink Me: (Ab)Using a LLM to Compress Text by mjburgess on Sunday May 05, 2024 @08:44AM (#40264449)
This is assuming the theory.

A statistical model of orbits, without a theory of gravity, is less compressed when you assume more objects. Take all the apparent positions of objects in the sky, {(object, x1, x2, t),...}. Find a statistical model of each point at t+1, so y = (o, x1, x2, t+1). There is no sense in which you're deriving a new object in the sky from this statistical model -- it is only a compression of observable orbits.

When you say, "if you have the new planet", you're changing the data generating process (theory) to produce a new distribution of points {(o' x1', x2', t'), ...} to include an unseen object. You're then comparing two data generating models (two theories) for their simplicity. You're not comparing the associative models.

Call the prior theory 8-planets, so 8P generates x1,x2,t; and the new theory 9P which generates x1',x2',t'

You're then making a conditional error distribution when comparing two rival theories. The 9P theory will minimize this error.

But in no sense can the 9P theory be derived from the initial associative statistical distribution. You are, based on theory (, science, knowledge, etc.) choosing to add a planet, vs. eg., correcting for measurement error, modifiying newton's laws, changing the angle of the earth wrt the solar system... or one of an infinite number of theories which all produce the same error minimization

The sense of "prediction" that science uses (via Popper et al.) is deriving the existence of novel phenomena that do not follow from prior observable distributions.

Re:Drink Me: (Ab)Using a LLM to Compress Text by mr_toad on Monday May 06, 2024 @07:44AM (#40273483)
> A statistical model of orbits, without a theory of gravity

You want a statistical model that produces a theory of gravity.

Re:Drink Me: (Ab)Using a LLM to Compress Text by Legend2440 on Sunday May 05, 2024 @12:05PM (#40265865)
It doesn’t matter how your predictor works, whether it’s a scientific theory, a statistical model, or a magic oracle. You can always perfectly convert its predictions into compression using entropy coding. The conversion process is black-box.
Re:Drink Me: (Ab)Using a LLM to Compress Text by mjburgess on Monday May 06, 2024 @02:35AM (#40271799)
yes, and this is a very trivial notion of prediction.

Predictions of novel objects derivative from scientific theories arent quantitative data points.