Re:The Limits of GPT-3 and Similar Large Language Models
by decremental on Saturday January 07, 2023 @03:08PM
(#34292125)
That might be too low(?) resolution. It would be learning encodings instead of features of the thing that is being encoded. Like training it on terabytes of zip files and expecting it to reproduce from the files contained in the archives.
[ Parent
| on HN ]
Re:The Limits of GPT-3 and Similar Large Language Models
by optimalsolver on Saturday January 07, 2023 @03:19PM
(#34292273)
Imagine a single, 120 minute movie in a tar file.
How much of this file's raw data would be encoding and metadata, vs the content of the movie?
Re:The Limits of GPT-3 and Similar Large Language Models
by decremental on Saturday January 07, 2023 @04:48PM
(#34293237)
The thing is even the video data outside of the tar file is also encoded. Most likely the compressed video data will be basically random. You can't train it on random data, it's just noise. Rather it would make more sense to train on sequences of RGBA pixels.
It's a seductive thought to be able to just throw raw bits at a model, regardless of what those bits represent, and have it just magically attain LLM qualities in reproducing the data you would want it to.
Something to think about: GPT3/ChatGPT tokenize at the byte level. If they tokenized at the bit level the model would learn Utf8 encoding over time. Unicode characters that require more than one byte to represent, such as emojis, are not learned directly but the model can still reproduce them.