News for hackers. Stuff that matters.
Re:Text Classification by Data Compression by thomasluce on Tuesday June 08, 2021 @06:50PM (#27442122)
I worked for an internet scraping/statistics gathering company some years ago, and we used this approach alongside a few others to find mailing addresses embedded in websites. Basically use LZW-type compression with entropy information only trained on known addresses, and then compress a document, looking for the section of the document with the highest compression ratio.

It worked decently well, and surprisingly better than a lot of other, more standard approaches just because of the wild non-uniformity of human-generated content on the web.

Re:Text Classification by Data Compression by ta988 on Tuesday June 08, 2021 @09:51PM (#27443337)
Does that mean you were doing an LZW compression but with a fixed table?
Re:Text Classification by Data Compression by thomasluce on Wednesday June 09, 2021 @01:25AM (#27444505)
Yes, exactly. We pre-built the table with a ton of hand-picked mailing addresses copy-pasted out of a bunch of free-text and then just kept using that one.
Re:Text Classification by Data Compression by ta988 on Wednesday June 09, 2021 @08:15PM (#27455160)
Great, I need to try something like that.