News for hackers. Stuff that matters.
Re:Text Classification by Data Compression by donatj on Tuesday June 08, 2021 @09:48PM (#27443324)
I work for an educational company and a proprietary metric we license includes how well the text gzips as one part of it's scoring system for reading difficulty.

The better it gzips, the easier it is to read.

It's just one part of the score, I'm not certain what weight it has on it.

Re:Text Classification by Data Compression by abriosi on Tuesday June 08, 2021 @10:08PM (#27443446)
Assuming the text is well written, from the standpoint of information theory, the better the text gzips the fewer information it contains
Re:Text Classification by Data Compression by jonnycomputer on Tuesday June 08, 2021 @11:57PM (#27444064)
Ah, well, this kind of information--entropy--isn't really the same thing as meaningful information (semantic information).

After all, take a dictionary. Lots of meaningful information there. But a lot less entropy than a random sequence of characters of the same length. That random sequence will have a lot more "information", like you said; its a real needle in the haystack, but its all haystack and no golden thread.

Re:Text Classification by Data Compression by abriosi on Tuesday June 08, 2021 @10:10PM (#27443463)
We could also argue that an understandable text has a high degree of redundancy, e.g. explaining the same concept with different perspectives
Re:Text Classification by Data Compression by tachyonbeam on Wednesday June 09, 2021 @12:11AM (#27444140)
I would say you can make your text easier to understand by a large audience by avoiding very rare words that many people don't understand, and sticking to common phrasings. That will make text compress better for sure. I'm sure you get literary brownie points for using 10 different words that mean pretty much the same thing in different contexts, but you should never do that in an engineering textbook, for instance.
Re:Text Classification by Data Compression by crznp on Tuesday June 08, 2021 @11:29PM (#27443907)
This comment only uses the ten hundred most used words and "xkcd" to give back love.

Writing that uses such a small set of words will probably be easy to make smaller when you save it. And for young students, it might help to have clear, simple writing because they don't have to learn new words too.

It might be too simple for older students (or people here). I would like to be able to name new ideas, and it sounds better if I don't have to use the same words over and over. The idea is the same, just relaxed a little bit. Give it a try, it makes you think about writing in a different way.

https://splasho.com/upgoer5/