Re:NNCP: Lossless Data Compression with Neural Networks (2019)
by TynanSigg on Saturday May 22, 2021 @04:50PM
(#27250149)
After a bit more digging, it seems Bellard also has a version using GPT-2: https://bellard.org/libnc/gpt2tc.html
Looks much cleaner and faster than my little winter break project, cool to see this implemented properly!
Re:NNCP: Lossless Data Compression with Neural Networks (2019)
by meiji163 on Saturday May 22, 2021 @03:55PM
(#27249771)
Using a pretrained model is a little bit cheating, you would have to include the size of the weights in the ratio
Re:NNCP: Lossless Data Compression with Neural Networks (2019)
by TynanSigg on Saturday May 22, 2021 @04:52PM
(#27250167)
Those two metrics will converge as the size of the text being compressed goes to infinity. It's necessary to include model size for things like the Hutter prize that involve compressing a fixed text (to avoid hard-coding) but isn't usually a useful metric for compression, especially because it will cause the compression ratio to depend on the size of the data being compressed.
Edit: I thought the model in the above paper was pre-trained, but apparently it's only trained on the data as it arrives! That's indeed a very interesting approach, I wouldn't have expected that neural network models would converge quickly enough for it to be useful.
Here's the code if anyone's interested: https://github.com/trsigg/lm-compression . I haven't looked at it in a while except for updating it today to work with more recent versions of the transformers library, but I think it should still function and is pretty easy to try out with different models. I thought about using something like a hidden Markov model instead which would be much faster (even more so because it would enable using tabled rather than range-based ANS), but I haven't gotten around to it.