Re:Dalai: Automatically install, run, and play with LLaMA on...
by gymbeaux on Sunday March 12, 2023 @09:02PM
(#35129137)
Everyone with “only” 64GB of RAM is pouting today, including me
[ Parent
| on HN ]
Re:Dalai: Automatically install, run, and play with LLaMA on...
by Taek on Sunday March 12, 2023 @09:05PM
(#35129178)
You can run llama using 4 bits per parameter, 64 GB of RAM is more than enough
Re:Dalai: Automatically install, run, and play with LLaMA on...
by geysersam on Monday March 13, 2023 @03:12AM
(#35132204)
4 bits is ridiculously little. I'm very curious what makes these models so robust to quantization.
Re:Dalai: Automatically install, run, and play with LLaMA on...
by MacsHeadroom on Monday March 13, 2023 @06:56AM
(#35133937)
Read The Case for 4 Bit Precision. https://arxiv.org/abs/2212.09720
Spoiler: it's the parameter count. As parameter count goes up, but depth matters less.
It just so happens that at around 10B+ parameters you can quantize down to 4bit with essentially no downsides. Models are that big now. So there's no need to waste RAM by having unnecessary precision for each parameter.
Re:Dalai: Automatically install, run, and play with LLaMA on...
by Taek on Monday March 13, 2023 @09:27AM
(#35135428)
For completeness, there's also another paper that demonstrated you get more power/accuracy per-bit at 4 bits than at any other level of precision (including 2 bits and 3 bits)
Re:Dalai: Automatically install, run, and play with LLaMA on...
by MacsHeadroom on Tuesday March 14, 2023 @12:38AM
(#35147506)
That's the paper I referenced. But newer research is already challenging it.
'Int-4 llama is not enough [0] - Int-3 and beyond' suggests 3-bit is best for models larger than ~10B parameters when combining binning and GPTQ.
[0] https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i...
Re:Dalai: Automatically install, run, and play with LLaMA on...
by metadat on Monday March 13, 2023 @12:21AM
(#35131018)
What if you have around 400GB of RAM? Would this be enough?