Re:Get answers for Shell Commands from GPT3 right from your ...
by moffkalast on Friday November 18, 2022 @06:41AM
(#33653274)
Sure, as long as you've got 350 GB of VRAM to load it.
[ Parent
| on HN ]
Re:Get answers for Shell Commands from GPT3 right from your ...
by visarga on Friday November 18, 2022 @11:39AM
(#33657279)
It's just 12B parameters, not 175B. You can probably use int16 or int8.
Re:Get answers for Shell Commands from GPT3 right from your ...
by moffkalast on Friday November 18, 2022 @07:33PM
(#33665171)
That's afaik already the size when using half precision. And it's not like GPT3 doesn't get half queries wrong as-is in its optimal form, detracting from it will just make it unusable. Check out GPT-J if you want to see a functionally useless model that can be run locally.
It's not like this is something that can be surmounted without extra storage, neural nets are just a fancy system for compression and data access. You can either go for less data or more compression but either way your results won't ever be as good.