News for hackers. Stuff that matters.
Re:Llama 2 – Meta AI by holoduke on Tuesday July 18, 2023 @01:37PM (#36776360)
So on a 4090 you cannot run the 70b model right?
Re:Llama 2 – Meta AI by pizza on Tuesday July 18, 2023 @02:14PM (#36776932)
You’d have to quantize the parameters to about 2.7 bits per parameter (24 GB / 70G * 8bits/B) - the model was likely trained at fp16 or fp32 so that would be pretty challenging. Not impossible but probably not readily available at the moment w most current quantization libraries. Quality would likely be degraded. But 2 4090s might be doable at ~4bits
Re:Llama 2 – Meta AI by nickolas_t on Tuesday July 18, 2023 @02:30PM (#36777172)
Sadly no, perhaps on a high end GPU in the year 2027(?)