News for hackers. Stuff that matters.
Re:The bigger-is-better approach to AI is running out of road by brucethemoose2 on Saturday June 24, 2023 @06:00PM (#36463037)
Its complicated.

Training on something huge like "the internet" is what gives rise to those amazing emergent properties missing in smaller models (including the recent Pi model). And there are only so many datasets that huge.

But its also indeed a waste, as Pi proves.

There probably is some sweet spot (6B-40B?) for specialized, heavily focused models pre trained with high quality general data.

There are no comments on this comment yet.