Bad numbers in the “gzip beats BERT” paper?

Re:Bad numbers in the “gzip beats BERT” paper? by ozr on Monday July 17, 2023 @10:39AM (#36758804)

The intuition about how the gzip method works goes like so:

If you compress `ABC`, it will be X bytes. If you then compress `ABCABC`, it will not take 2x bytes. The more similar the two strings that you concatenate, the less bytes it will take. `ABCABD` will take more than `ABCABC`, but less than `ABCXYZ`.

BERT is, by todays standards, a very small LLM, which we know has weaker performance than the billion-param scale models most of us are interacting with today.

[ Parent | on HN ]

Re:Bad numbers in the “gzip beats BERT” paper? by jabbany on Monday July 17, 2023 @10:49AM (#36758938)

> very small LLM

Heh. So does that make it a MLM (medium)?

I've always found it funny that we've settled on a term for a class of models that has a size claim... Especially given how fast things are evolving...

[ Parent | on HN ]