News for hackers. Stuff that matters.
Re:Ts_zip: Text Compression Using Large Language Models by optimalsolver on Wednesday August 16, 2023 @04:18PM (#37153139)
Reminder that compression = intelligence:

https://mattmahoney.net/dc/rationale.html

...which leads to the theory of mind known as "compressionism":

https://ceur-ws.org/Vol-1419/paper0045.pdf

Re:Ts_zip: Text Compression Using Large Language Models by tysam_and on Wednesday August 16, 2023 @04:41PM (#37153484)
No. Compression == information, not intelligence. This is a common mistake that people make, and is a popular expression making the rounds right now, but it is incorrect.

Intelligence _may arise_ from information. Information _necessarily arises_ from compression.

It is very similar, to me, to the concept of humors. Empirically, it was observed that these things had some correlation, and they occurred together, but they are not directly causative. Similarly with the theory of spontaneous generation (https://en.wikipedia.org/wiki/Spontaneous_generation), which was another theory spawned (heh, as it were), from casual causal correlation (again, as it were).

This can be shown rather easily when you show that the time-dynamics of an exemplar system with high information diverge from the time dynamics of a system with high intelligence, i.e. there is a structural component to the information embedded in the system such that the temporal aspects of applied information are somehow necessary for intelligence.

I hope to release some work at least tangentially related to this within the next few years (though of course it is a bit more high-flying and depends on some other in-flight work). If we are to attempt to move towards AGI, I really think we need to stick with the mathematical basics. Neural networks, although they've been made out to be really complicated, are actually quite simple in terms of the concepts powering them, I believe.

Re:Ts_zip: Text Compression Using Large Language Models by bbstats on Wednesday August 16, 2023 @05:38PM (#37154219)
OTOH true intelligence can still poorly compress things.
Re:Ts_zip: Text Compression Using Large Language Models by tysam_and on Wednesday August 16, 2023 @05:39PM (#37154233)
Any ERM-based method will have this flaw (if it is a flaw -- I'd possibly consider it a tradeoff as well) -- it is the nature of compression itself. I believe this disrupts the 'information' layer of the hierarchy, not the 'intelligence', though that is just my personal 2 cents of course.
Re:Ts_zip: Text Compression Using Large Language Models by derefr on Wednesday August 16, 2023 @06:39PM (#37154836)
I think what the GP meant is not that compressed data has intelligence, but rather that a compressor is necessarily "intelligent" in correlation to the degree that it compresses well on arbitrary novel datasets it wasn't tuned for. At least insofar as we define "intelligence" with measures like IQ, that relate to "number of examples of a pattern required to notice the pattern."

To compress data, you have to derive some interesting insight about that data that allows the data to be projected into a lower-dimensional, lower-entropy embedding.

The more things a compressor can do that for — without the compressor itself being made larger — the more "intelligent" the compressor is, no?

Re:Ts_zip: Text Compression Using Large Language Models by tysam_and on Wednesday August 16, 2023 @07:12PM (#37155108)
Yes, that is the point I'm addressing, I'm using Shannon's definition of information and making an argument about the _temporal_ aspects of it. If you're looking for a hint, it's in the Lyapunov exponents of the system, which do not come necessarily from compression alone, though compression I feel is a critical part of the process.

Hence, the set vs superset argument.

Re:Ts_zip: Text Compression Using Large Language Models by rurban on Friday August 18, 2023 @11:24AM (#37176885)
No, compressability == complexity.

Kolmogorov actually

Re:Ts_zip: Text Compression Using Large Language Models by tysam_and on Saturday August 19, 2023 @06:15PM (#37193858)
To be a little blunt, this comment does not make that much sense.

Kolmogorov complexity is based directly off of Shannon's information theory. It's the definition of the shortest possible length of a program that can output a particular string.

It is an extension of information theory. If you just take either of the words 'Kolmogorov' or 'complexity' out of the phrase 'Kolmogorov complexity', it loses all meaning. We measure the programs for a given string against that string's Kolmogorov complexity by their entropy. Hence, why at the core, the very pith of the matter at hand is information, which is a measurable quantity. Additionally, K complexity is a lower bound, not a quantity.

This is a similar kind of swap to saying "No, those baked goods are not food, because goods == items bought and sold for currency", if this recontextualizes it a bit.

Re:Ts_zip: Text Compression Using Large Language Models by ks2048 on Wednesday August 16, 2023 @04:30PM (#37153345)
I would say compression and intelligence are closely related, not equal. For example, zstd is a pretty good compressor, but does any one think it should be called a pretty good “AI”?
Re:Ts_zip: Text Compression Using Large Language Models by dragonwriter on Wednesday August 16, 2023 @04:34PM (#37153398)
Wasn't there a paper recently about a fairly simple wrapper around a common conpression algorithm beating LLMs on classification tasks?
Re:Ts_zip: Text Compression Using Large Language Models by jmholla on Wednesday August 16, 2023 @04:49PM (#37153599)
I thought that was about LLMs being trained on compressed data. But I might be thinking about a different paper.
Re:Ts_zip: Text Compression Using Large Language Models by dragonwriter on Wednesday August 16, 2023 @04:54PM (#37153684)
Gzip + kNN for text classification:

https://aclanthology.org/2023.findings-acl.426.pdf

Re:Ts_zip: Text Compression Using Large Language Models by junipertea on Wednesday August 16, 2023 @05:26PM (#37154082)
Not LLM, just BERT, also did not actually outperform it.

source: https://kenschutte.com/gzip-knn-paper/

Re:Ts_zip: Text Compression Using Large Language Models by willis936 on Wednesday August 16, 2023 @05:09PM (#37153878)
Someone in sales with a product that uses zstd.
Re:Ts_zip: Text Compression Using Large Language Models by londons_explore on Wednesday August 16, 2023 @05:37PM (#37154206)
Read the paper...

Zstd, while it might be better than gzip or bzip, is still a very poor compressor compared to an ideal compressor (which hasn't yet been discovered).

That is why zstd acts like a rather bad AI. Note that if you wanted to use zstd as an AI, you would patch out of the source code checksum checks, and you would then feed it a file to decompress (The cat sat on the mat), followed by random noise.

A great compressor would output: The cat sat on the mat. It was comfortable, so he then lay down to sleep.

A medium compressor would output: The cat sat on the mat. bat cat cat mat sat bat.

A terrible compressor would output: The cat sat on the mat. Dsr %we3 9T23 }£{D:rg!@ !jv£dP$

Re:Ts_zip: Text Compression Using Large Language Models by astrange on Wednesday August 16, 2023 @05:52PM (#37154383)
> A terrible compressor would output: The cat sat on the mat. Dsr %we3 9T23 }£{D:rg!@ !jv£dP$

LLMs are sort of unable to do this because they use a fixed tokenizer instead of raw bytes. That means they won't output binary garbage even early on + saves a lot of memory, but it may hurt learning things like capitalization, rhyming, etc we think are obvious.

Re:Ts_zip: Text Compression Using Large Language Models by londons_explore on Wednesday August 16, 2023 @06:02PM (#37154480)
Even if you trained an LLM with a simplified tokenizer that simply had 256 tokens for each of 256 possible ascii characters, you would see the same result.
Re:Ts_zip: Text Compression Using Large Language Models by canjobear on Wednesday August 16, 2023 @11:21PM (#37156751)
Concretely gzip will output something like "theudcanvas. ;cm,zumhmcyoetter toauuo long a one aay,;wvbu.mvns. x the dtls and enso.;k.like bla.njv"

https://github.com/Futrell/ziplm

Re:Ts_zip: Text Compression Using Large Language Models by twitch_checksum on Thursday August 17, 2023 @10:13PM (#37170723)
Well, Zstandard can be used as a very fast classification engine, which is a task traditionally done by AI : https://twitter.com/abhi9u/status/1683141215871705088 https://github.com/cyrilou242/ftcc
Re:Ts_zip: Text Compression Using Large Language Models by burtonator on Wednesday August 16, 2023 @04:46PM (#37153548)
There must be a relationship here between Shannon's estimation that english is 1 bit of entropy per character and is highly redundant and 'easy' to predict.

A highly advanced AI could compress the text and predict the next sequences easily.

This seems like a direct connection like electricity and magnetism.

And maybe that's why English needs to be about 1 bit because we're not very intelligent.

Re:Ts_zip: Text Compression Using Large Language Models by YeGoblynQueenne on Wednesday August 16, 2023 @04:48PM (#37153591)
>> Reminder that compression = intelligence:

If I conclude from this that you mean that gzip is intelligent, will I be accused of bad faith?

Re:Ts_zip: Text Compression Using Large Language Models by bbstats on Wednesday August 16, 2023 @05:36PM (#37154194)
I would say compression has some congruency with intelligence.
Re:Ts_zip: Text Compression Using Large Language Models by canjobear on Wednesday August 16, 2023 @11:22PM (#37156764)
Compression is equivalent to learning but it's not clear to me that that gets you all the way to intelligent action. In reinforcement learning terms, it can get you asymptotically to a perfect model of the environment, but it's not clear to me that it would tell you how to find the highest-value action in that environment.