Alpaca- An Instruct Tuned Llama 7B. Responses on par with txt-DaVinci-3. Demo up

Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by notpushkin on Monday March 13, 2023 @11:55PM (#35147249)

What's the correlation between parameter count and RAM usage? Will LLaMA-13B fit on my MacBook Air with 8 GB of RAM or am I stuck with 7B?

[ Parent | on HN ]

Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by maxxk on Tuesday March 14, 2023 @01:07AM (#35147647)

13B uses about 9GB on my MacBook Air. If you have another machine (x86) with enough RAM to convert the original LLaMA representation to GGML, you can give it a try. But quantization step must be done on MacBook.

Maybe it is more feasible for you to use 7B with larger context. For some "autocompletion" experiments with Python code I had to extend context to 2048 tokens (+1-1.5GB).

[ Parent | on HN ]