Re:Dalai: Automatically install, run, and play with LLaMA on...
by boredemployee on Sunday March 12, 2023 @08:42PM
(#35128931)
sorry for the extremely dumb question but is it possible to run the 68B model in a 8gb ram computer?
[ Parent
| on HN ]
Re:Dalai: Automatically install, run, and play with LLaMA on...
by infinityio on Sunday March 12, 2023 @09:52PM
(#35129698)
in general, assume 2GB per billion parameters - with quantisation you can get this down to <1GB (~500MB for 3 bit?), but even with that you'll only be able to run quantised llama-13B in the best case
Having said that: if you are feeling incredibly patient you can technically run the 68B parameter model by swapping to disk, although it will not be a pleasant experience (think minutes or hours per token instead of tokens per second)
Additionally worth noting pure CPU inference is much slower than GPU/TPU inference, so the output will be much slower than a ChatGPT-like service even if it does fit in your computer's RAM
Re:Dalai: Automatically install, run, and play with LLaMA on...
by boredemployee on Sunday March 12, 2023 @10:51PM
(#35130349)
thanks for explaining! How much GPU memory would work nice with 68B?
Re:Dalai: Automatically install, run, and play with LLaMA on...
by ukd1 on Monday March 13, 2023 @01:03AM
(#35131317)
they said 2g per 1 billion....and it's called 68B...I presume that's 68 billion... 68*2...so at least 136g?
Re:Dalai: Automatically install, run, and play with LLaMA on...
by vishal0123 on Monday March 13, 2023 @04:37AM
(#35132789)
68/2, not 682
Re:Dalai: Automatically install, run, and play with LLaMA on...
by boredemployee on Monday March 13, 2023 @05:16AM
(#35133100)
So, if I understand correctly, that's what you need to run the best model?
With GPU:
VRAM + RAM >= 68/2
Without GPU:
RAM >= 68/2
Re:Dalai: Automatically install, run, and play with LLaMA on...
by coldtea on Monday March 13, 2023 @05:40AM
(#35133318)
Not sure about the "=" part. You'd want some memory for the compositor and other OS graphics, and regular RAM for OS and programs, no?
Re:Dalai: Automatically install, run, and play with LLaMA on...
by Tepix on Monday March 13, 2023 @12:29PM
(#35138026)
You can't, it needs around 40GB of RAM.
Technically you can by swapping to disk but it would be too slow to be usable.
What you can do however is use the 7B model with 4bit quantization and use it within 8GB RAM.