News for hackers. Stuff that matters.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by atylerrice on Tuesday February 14, 2023 @10:03PM (#34799522)
You can also use int8 quantization which comes for free with huggingface and isn’t a huge cost to accuracy. Which cuts the memory usage almost in half again. (almost in half because some layers might stay 32 bit if they make a tiny part of the model)

There are no comments on this comment yet.