Re:LLaMA: A foundational, 65B-parameter large language model
by blablablub on Friday February 24, 2023 @01:19PM
(#34927817)
So what hardware do we need to run this model?
[ Parent
| on HN ]
Re:LLaMA: A foundational, 65B-parameter large language model
by CuriouslyC on Friday February 24, 2023 @02:20PM
(#34928721)
7 billion can run on 16+ gb GPUs as fp16, 14 billion can be run on 16+ gb if quantized to int8. 14G @ fp16 and 30G at int8 will require one of the 48 gb cards (less, but hardware mostly goes 24 -> 48).
Re:LLaMA: A foundational, 65B-parameter large language model
by blablablub on Friday February 24, 2023 @04:03PM
(#34930095)
thanks
Re:LLaMA: A foundational, 65B-parameter large language model
by brucethemoose2 on Friday February 24, 2023 @04:58PM
(#34930713)
Requirements could be reduced with something like DeepSpeed or ColossalAI (or even just simple hacks to move bits to RAM more aggressively)