Re:Llama 2 – Meta AI
by pizza on Tuesday July 18, 2023 @02:14PM
(#36776932)
You’d have to quantize the parameters to about 2.7 bits per parameter (24 GB / 70G * 8bits/B) - the model was likely trained at fp16 or fp32 so that would be pretty challenging. Not impossible but probably not readily available at the moment w most current quantization libraries. Quality would likely be degraded. But 2 4090s might be doable at ~4bits
[ Parent
| on HN ]
There are no comments on this comment yet.