News for hackers. Stuff that matters.
Re:OpenAI Leaks 120B Open Model on Hugging Face by sciencesama on Friday August 01, 2025 @01:42PM (#44759916)
How much ram do you need to run this !!??
Re:OpenAI Leaks 120B Open Model on Hugging Face by cubefox on Friday August 01, 2025 @02:41PM (#44760725)
Probably about one byte per weight (parameter) plus a bit extra for the key-value cache (depends on the size of the context window).
Re:OpenAI Leaks 120B Open Model on Hugging Face by int_19h on Friday August 01, 2025 @03:32PM (#44761350)
You can go below one byte per parameter. 4-bit quantization is fairly popular. It does affect quality - for some models more so than others - but, generally speaking, a 4-bit quantized model is still going to do significantly better than an 8-bit model with 1/2 parameters.