Re:OpenAI Leaks 120B Open Model on Hugging Face
by cubefox on Friday August 01, 2025 @02:41PM
(#44760725)
Probably about one byte per weight (parameter) plus a bit extra for the key-value cache (depends on the size of the context window).
[ Parent
| on HN ]
Re:OpenAI Leaks 120B Open Model on Hugging Face
by int_19h on Friday August 01, 2025 @03:32PM
(#44761350)
You can go below one byte per parameter. 4-bit quantization is fairly popular. It does affect quality - for some models more so than others - but, generally speaking, a 4-bit quantized model is still going to do significantly better than an 8-bit model with 1/2 parameters.