News for hackers. Stuff that matters.
Re:Adventures with Compression by xnx on Saturday December 30, 2023 @11:14PM (#38821728)
> GPT has this information encoded implicitly in its neural weights in a much more sophisticated way than linguistic theories can achieve

"More data beats clever algorithms" strikes again.

Re:Adventures with Compression by bawolff on Sunday December 31, 2023 @12:59AM (#38822165)
Isn't gpt like 800gb big? I can compress a 1gb file to 0 bytes if i am allowed 1gb of additional data.
Re:Adventures with Compression by ChadNauseam on Sunday December 31, 2023 @01:09AM (#38822200)
You can't if you have to pick the 1gb of additional data before you see the 1gb file you're supposed to compress (which is the situation gpt is in)
Re:Adventures with Compression by bawolff on Sunday December 31, 2023 @02:34AM (#38822463)
Although in this case, we know the data comes from wikipedia and i assume gpt was trained on wikipedia, so it really does seem unfair.
Re:Adventures with Compression by _a_a_a_ on Sunday December 31, 2023 @07:30AM (#38823616)
But the parent was talking about lossless, GPT presumably is anything but.
Re:Adventures with Compression by pizza on Sunday December 31, 2023 @08:07AM (#38823794)
Sure it is, just use a probability distribution over the entire vocabulary of possible tokens to do arithmetic/range coding of the original text.
Re:Adventures with Compression by _a_a_a_ on Sunday December 31, 2023 @09:06AM (#38824101)
Uh? GPT is an neural net, right?Which is a bunch of weights, surely, so how can you get lossless compression from that?

(Complete newbie here BTW)

Re:Adventures with Compression by pizza on Sunday December 31, 2023 @09:26AM (#38824255)
The model can either generate probable outputs or score the likelihood of given inputs.
Re:Adventures with Compression by _a_a_a_ on Sunday December 31, 2023 @10:43AM (#38824795)
How does one produce 'lossless' from 'probable' or 'likelihood'?
Re:Adventures with Compression by qazxcvbnm on Sunday December 31, 2023 @12:30PM (#38825729)
Arithmetic coding is one of the classic information theory methods that produce essentially-optimal results. Like everywhere else in information theory, the eminence of the probability distribution is fully recognised, and given it, arithmetic coding gives you an information-theoretically optimal lossless compression (to the degree that the probability distribution describes the data well) (by treating the objects in question as numbers and intuitively, like the other post says, only keeping diffs).

See https://en.wikipedia.org/wiki/Arithmetic_coding

Re:Adventures with Compression by sp332 on Sunday December 31, 2023 @11:06AM (#38824975)
By running the decoder during encoding. If the decoder already produces the right output, which it will most of the time, then you don’t need to store anything in the file. If it produces the wrong output, store a correction, then continue from there. The final file only needs to include the corrections.
Re:Adventures with Compression by pizza on Sunday December 31, 2023 @12:46PM (#38825902)
Let's say your text is "I like red cats", your current token input is at "I like red". Let's say the model thinks "cats" is 25% likely. It also thinks "dogs" is eg 50% likely for the next token. So rather than storing 1 bit (= log p) to encode dog in the output encoded code, it can use 2 bits to store "cats". Totally lossless. At decode time, you're reading the code, so it's going to just pick the "cats" token anyway, even though it's not the likeliest, because it's not randomly sampling here, it's just using the underlying probabilistic model to do entropy coding. It's not lossy here, since we're not letting the model/sampling process freely choose what the text is made of.
Re:Adventures with Compression by d_burfoot on Tuesday January 02, 2024 @10:22AM (#38842529)
GPT, and probably any LLM, can be hooked up to an encoder to produce a lossless data compressor. Patrice Bellard has done this, HN discussion here:

https://news.ycombinator.com/item?id=23618465