News for hackers. Stuff that matters.

A comment on OpenAI's justification for why training data is fair use, not infringement [pdf]. Parent comment

Re:OpenAI's justification for why training data is fair use,... by Zuiii on Thursday October 05, 2023 @05:00PM (#37784239)
These models are generally trained on tarabytes of data, but are usually 10s of gigabytes large (or much less if quantized). The latest true open source model, mistral 7b, is only 3GB (0.3% of a TB) when quantized.
[ Parent | on HN ]
Re:OpenAI's justification for why training data is fair use,... by rpdillon on Thursday October 05, 2023 @07:28PM (#37785527)
I did a very similar analysis with Llama 65B being trained on 5.6T tokens assuming token length of 4 characters and comparing with a quantized model size of ~38GB.

The 3% number was a conservative rounding of the same calculation, but retaining fp16 rather than quantizing to 4 bits.

Here's my original back of the napkin analysis:

https://news.ycombinator.com/item?id=36681440

[ Parent | on HN ]