Re:A comprehensive guide to running Llama 2 locally
by lolinder on Tuesday July 25, 2023 @04:11PM
(#36868639)
Depends on which model. I haven't bothered doing it on my 8GB because the only model that would fit is the 7B model quantized to 4 bits, and that model at that size is pretty bad for most things. I think you could have fun with 13B with 12GB VRAM. The full size model would require >35GB even quantized.
[ Parent
| on HN ]
There are no comments on this comment yet.