News for hackers. Stuff that matters.
Re:Get answers for Shell Commands from GPT3 right from your ... by visarga on Friday November 18, 2022 @11:39AM (#33657279)
It's just 12B parameters, not 175B. You can probably use int16 or int8.
Re:Get answers for Shell Commands from GPT3 right from your ... by moffkalast on Friday November 18, 2022 @07:33PM (#33665171)
That's afaik already the size when using half precision. And it's not like GPT3 doesn't get half queries wrong as-is in its optimal form, detracting from it will just make it unusable. Check out GPT-J if you want to see a functionally useless model that can be run locally.

It's not like this is something that can be surmounted without extra storage, neural nets are just a fancy system for compression and data access. You can either go for less data or more compression but either way your results won't ever be as good.