News for hackers. Stuff that matters.
Re:Llama 2 – Meta AI by pizza on Tuesday July 18, 2023 @02:14PM (#36776932)
You’d have to quantize the parameters to about 2.7 bits per parameter (24 GB / 70G * 8bits/B) - the model was likely trained at fp16 or fp32 so that would be pretty challenging. Not impossible but probably not readily available at the moment w most current quantization libraries. Quality would likely be degraded. But 2 4090s might be doable at ~4bits

There are no comments on this comment yet.