News for hackers. Stuff that matters.

A comment on The bigger-is-better approach to AI is running out of road. Parent comment

Re:The bigger-is-better approach to AI is running out of road by hooande on Sunday June 25, 2023 @12:16AM (#36465095)
Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. Any individual reddit thread is an atom in a drop in a bucket. All of reddit is < 2% of the total training data, regardless of quality.

Yes, the correct thing to do is get more data. Much more.

[ Parent | on HN ]
Re:The bigger-is-better approach to AI is running out of road by ftxbro on Sunday June 25, 2023 @03:46AM (#36465870)
> Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data.

It matters a little bit, in a quantitative but not qualitative way. Probably with good data cleaning you could get as high quality result with only one pebibyte of data if it normally needs two pebibytes. If training time is proportional to dataset size then maybe it takes three months instead of six months to train. Maybe it would save hundreds of millions or a billion dollars which I guess would matter to someone. It probably wouldn't matter qualitatively though.

[ Parent | on HN ]
Re:The bigger-is-better approach to AI is running out of road by imtringued on Sunday June 25, 2023 @07:17AM (#36467026)
Nothing like bruteforcing intelligence by simply building a large enough lookup table to contain responses to all potential questions.
[ Parent | on HN ]