News for hackers. Stuff that matters.
Re:Ts_zip: Text Compression Using Large Language Models by ks2048 on Wednesday August 16, 2023 @04:30PM (#37153345)
I would say compression and intelligence are closely related, not equal. For example, zstd is a pretty good compressor, but does any one think it should be called a pretty good “AI”?
Re:Ts_zip: Text Compression Using Large Language Models by dragonwriter on Wednesday August 16, 2023 @04:34PM (#37153398)
Wasn't there a paper recently about a fairly simple wrapper around a common conpression algorithm beating LLMs on classification tasks?
Re:Ts_zip: Text Compression Using Large Language Models by jmholla on Wednesday August 16, 2023 @04:49PM (#37153599)
I thought that was about LLMs being trained on compressed data. But I might be thinking about a different paper.
Re:Ts_zip: Text Compression Using Large Language Models by dragonwriter on Wednesday August 16, 2023 @04:54PM (#37153684)
Gzip + kNN for text classification:

https://aclanthology.org/2023.findings-acl.426.pdf

Re:Ts_zip: Text Compression Using Large Language Models by junipertea on Wednesday August 16, 2023 @05:26PM (#37154082)
Not LLM, just BERT, also did not actually outperform it.

source: https://kenschutte.com/gzip-knn-paper/

Re:Ts_zip: Text Compression Using Large Language Models by willis936 on Wednesday August 16, 2023 @05:09PM (#37153878)
Someone in sales with a product that uses zstd.
Re:Ts_zip: Text Compression Using Large Language Models by londons_explore on Wednesday August 16, 2023 @05:37PM (#37154206)
Read the paper...

Zstd, while it might be better than gzip or bzip, is still a very poor compressor compared to an ideal compressor (which hasn't yet been discovered).

That is why zstd acts like a rather bad AI. Note that if you wanted to use zstd as an AI, you would patch out of the source code checksum checks, and you would then feed it a file to decompress (The cat sat on the mat), followed by random noise.

A great compressor would output: The cat sat on the mat. It was comfortable, so he then lay down to sleep.

A medium compressor would output: The cat sat on the mat. bat cat cat mat sat bat.

A terrible compressor would output: The cat sat on the mat. Dsr %we3 9T23 }£{D:rg!@ !jv£dP$

Re:Ts_zip: Text Compression Using Large Language Models by astrange on Wednesday August 16, 2023 @05:52PM (#37154383)
> A terrible compressor would output: The cat sat on the mat. Dsr %we3 9T23 }£{D:rg!@ !jv£dP$

LLMs are sort of unable to do this because they use a fixed tokenizer instead of raw bytes. That means they won't output binary garbage even early on + saves a lot of memory, but it may hurt learning things like capitalization, rhyming, etc we think are obvious.

Re:Ts_zip: Text Compression Using Large Language Models by londons_explore on Wednesday August 16, 2023 @06:02PM (#37154480)
Even if you trained an LLM with a simplified tokenizer that simply had 256 tokens for each of 256 possible ascii characters, you would see the same result.
Re:Ts_zip: Text Compression Using Large Language Models by canjobear on Wednesday August 16, 2023 @11:21PM (#37156751)
Concretely gzip will output something like "theudcanvas. ;cm,zumhmcyoetter toauuo long a one aay,;wvbu.mvns. x the dtls and enso.;k.like bla.njv"

https://github.com/Futrell/ziplm

Re:Ts_zip: Text Compression Using Large Language Models by twitch_checksum on Thursday August 17, 2023 @10:13PM (#37170723)
Well, Zstandard can be used as a very fast classification engine, which is a task traditionally done by AI : https://twitter.com/abhi9u/status/1683141215871705088 https://github.com/cyrilou242/ftcc