Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool
by atylerrice on Tuesday February 14, 2023 @10:03PM
(#34799522)
You can also use int8 quantization which comes for free with huggingface and isn’t a huge cost to accuracy. Which cuts the memory usage almost in half again. (almost in half because some layers might stay 32 bit if they make a tiny part of the model)
[ Parent
| on HN ]
There are no comments on this comment yet.