Re:Using LLaMA with M1 Mac and Python 3.11
by trillic on Sunday March 12, 2023 @02:11PM
(#35123867)
So you’re saying I could make the full model run on a 16 core ryzen with 64GB of DDR4? I have an 8GB VRAM 3070 but based on this thread it sounds like the CPU might have better perf due to the RAM?
[ Parent
| on HN ]
Re:Using LLaMA with M1 Mac and Python 3.11
by sebzim4500 on Sunday March 12, 2023 @02:25PM
(#35124073)
These are my observations from playing with this over the weekend.
1. There is no thoughput benefit to running on GPU unless you can fit all the weights in VRAM. Otherwise the moving the weights eats up any benefit you can get from the faster compute.
2. The quantized models do worse than non-quantized smaller models, so currently they aren't worth using for much use cases. My hope is that more sophisticated quantization methods (like GPTQ) will resolve this.
3. Much like using raw GPT-3, you need to put a lot of thought into your prompts. You can really tell it hasn't been 'aligned' or whatever the kids are calling it these days.
Re:Using LLaMA with M1 Mac and Python 3.11
by blablablub on Monday March 13, 2023 @04:11PM
(#35141793)
i have the 65B model running fine on my 48GB Ryzen 5.