Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by shad0wca7 on Saturday March 11, 2023 @07:26AM (#35107733)

Since the quantization requires more ram than running it, why can’t the quantized models be uploaded for use by those with 16-32gb?

[ Parent | on HN ]

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by sp332 on Saturday March 11, 2023 @09:09AM (#35108395)

It's against the license terms, but seeing how quickly the weights were leaked in the first place, I wouldn't be surprised if a torrent shows up with quantized ones soon.

[ Parent | on HN ]

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by MacsHeadroom on Saturday March 11, 2023 @10:07PM (#35116050)

They can and they have been look on HuggingFace or search torrent sites.

You only actually need about 30GB of VRAM (or unified memory) and no ram to run the largest 65B model.

[ Parent | on HN ]