News for hackers. Stuff that matters.

A comment on Hutter Prize: Compress a 100MB file to less than the current record of 16 MB. Parent comment

Re:Hutter Prize: Compress a 100MB file to less than the curr... by gwern on Sunday August 11, 2019 @05:08PM (#20670524)
All something like GPT-2 is, is a text compressor, a model for an arithmetic encoder specifically. It predicts the probability of the next token conditional on a history. You can directly measure the 'bits per character'/'perplexity' as a measure of (compression) performance, and that is typically how these language models are evaluated: eg http://nlpprogress.com/english/language_modeling.html https://paperswithcode.com/task/language-modelling

The Hutter dataset is included, incidentally, and the top is a Transformer-XL (277M parameters) at 0.94 BPC. Does all of that seem 'very much not AGI'?

[ Parent | on HN ]
Re:Hutter Prize: Compress a 100MB file to less than the curr... by tux3 on Sunday August 11, 2019 @05:19PM (#20670592)
What's unfortunate is the small size of enwiki8.

A decompressor with a high upfront cost (say a pretrained model) and otherwise higher compression ratio would be more interesting than tweaking yet another PAQ8 variant, but I'm not sure this contest's rules make that very viable.

[ Parent | on HN ]
Re:Hutter Prize: Compress a 100MB file to less than the curr... by doctorpangloss on Sunday August 11, 2019 @05:11PM (#20670549)
Yes, it does not at all seem like AGI! You’ve explained succinctly why the GPT2 result is maybe not on the road to AGI.
[ Parent | on HN ]
Re:Hutter Prize: Compress a 100MB file to less than the curr... by gwern on Sunday August 11, 2019 @05:30PM (#20670655)
Language models trained in a predictive fashion achieve ever closer to human compression rates (humans would only get ~0.7-0.8 BPC on Hutter), and are responsible for not just the creepiest & most realistic text generation to date but also setting SOTAs across all language-related AI benchmarks like SuperGLUE, designed to test language understanding and real-world reasoning, or chatbots. If that has nothing whatsoever to do with progress towards AGI, we'll just have to agree to disagree.
[ Parent | on HN ]
Re:Hutter Prize: Compress a 100MB file to less than the curr... by dwohnitmok on Sunday August 11, 2019 @05:36PM (#20670688)
I found Scott Alexander's take on this fairly convincing. Essentially if you master language you get broadly generalizable insights across domains.

https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...

If you think of intelligence as distilling the essence of the universe around you and building on top of that, at least the former starts to sound like really really really sophisticated compression.

[ Parent | on HN ]