News for hackers. Stuff that matters.

A comment on How Is LLaMa.cpp Possible?.

Re:How Is LLaMa.cpp Possible? by m3kw9 on Tuesday August 15, 2023 @08:33PM (#37141250)
Is there a Llama2 65b quantized version for Mac M2?
[ Parent | on HN ]
Re:How Is LLaMa.cpp Possible? by eurekin on Tuesday August 15, 2023 @09:07PM (#37141514)
I'd look for llama2 thebloke 70b GGML on hugginface.

Update, this might work: https://huggingface.co/TheBloke/Llama-2-70B-GGML

Example command mentions the 4bit variant:

./main -m llama-2-70b.ggmlv3.q4_0.bin -gqa 8 -t 13 -p "Llamas are"

[ Parent | on HN ]
Re:How Is LLaMa.cpp Possible? by ionwake on Wednesday August 16, 2023 @06:14AM (#37144958)
I also found this link ( https://blog.lastmileai.dev/run-llama-2-locally-in-7-lines-a...) but havent verified if it helps, but thanks for your reply I am taking a look now.
[ Parent | on HN ]
Re:How Is LLaMa.cpp Possible? by eurekin on Thursday August 17, 2023 @01:24PM (#37165162)
Just checked on a m1 w 32GB ram. 7b and 13b work as a charm. 70b doesn't fit, even the 2 bit version
[ Parent | on HN ]
Re:How Is LLaMa.cpp Possible? by orost on Tuesday August 15, 2023 @09:23PM (#37141635)
https://huggingface.co/TheBloke/Llama-2-70B-GGML/tree/main

Pick your quantization level depending on how much memory you have and how much speed you need

Run with llama.cpp (or kobold.cpp if you want a convenient webui)

(There is no llama2 65B, it's 70B instead)

[ Parent | on HN ]