News for hackers. Stuff that matters.

A comment on Facebook LLAMA is being openly distributed via torrents. Parent comment

Re:Facebook LLAMA is being openly distributed via torrents by mlboss on Friday March 03, 2023 @09:55AM (#35010367)
Is it possible to run the smallest one on a consumer gpu with 24gb ram ?
[ Parent | on HN ]
Re:Facebook LLAMA is being openly distributed via torrents by rihegher on Friday March 03, 2023 @10:01AM (#35010452)
I would be surprised if you can't. The smallest weight file is 14gb apparently
[ Parent | on HN ]
Re:Facebook LLAMA is being openly distributed via torrents by eightysixfour on Friday March 03, 2023 @12:43PM (#35012761)
https://github.com/facebookresearch/llama/blob/main/FAQ.md#3

Looks like it needs 14gb for weights and it isn't clear what the minimum size for the decoding cache is, but it defaults to settings for 30gb GPUs.

[ Parent | on HN ]
Re:Facebook LLAMA is being openly distributed via torrents by MacsHeadroom on Saturday March 04, 2023 @09:30PM (#35026444)
In int8 7B needs only 9GB of VRAM and 13B needs only 20GB on a single GPU. https://github.com/oobabooga/text-generation-webui/issues/14...
[ Parent | on HN ]
Re:Facebook LLAMA is being openly distributed via torrents by MacsHeadroom on Saturday March 04, 2023 @09:29PM (#35026441)
You can do even better!. You can run the second smallest one (better than GPT-3 175B) on 24GB of vram, ie LLaMA-13B. https://github.com/oobabooga/text-generation-webui/issues/14...
[ Parent | on HN ]
Re:Facebook LLAMA is being openly distributed via torrents by Tepix on Monday March 06, 2023 @01:45PM (#35045647)
Running it is easy but you'll probably want to finetune it, too
[ Parent | on HN ]