Facebook LLAMA is being openly distributed via torrents

Re:Facebook LLAMA is being openly distributed via torrents by aent on Friday March 03, 2023 @06:56AM (#35008366)

For anyone wondering, it includes 4 models: 7/13/30/65 billion parameters, the smallest one is 14Gb, the largest one is 131GB, all four are 235Gb.

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by mlboss on Friday March 03, 2023 @09:55AM (#35010367)

Is it possible to run the smallest one on a consumer gpu with 24gb ram ?

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by rihegher on Friday March 03, 2023 @10:01AM (#35010452)

I would be surprised if you can't. The smallest weight file is 14gb apparently

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by eightysixfour on Friday March 03, 2023 @12:43PM (#35012761)

https://github.com/facebookresearch/llama/blob/main/FAQ.md#3

Looks like it needs 14gb for weights and it isn't clear what the minimum size for the decoding cache is, but it defaults to settings for 30gb GPUs.

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by MacsHeadroom on Saturday March 04, 2023 @09:30PM (#35026444)

In int8 7B needs only 9GB of VRAM and 13B needs only 20GB on a single GPU. https://github.com/oobabooga/text-generation-webui/issues/14...

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by MacsHeadroom on Saturday March 04, 2023 @09:29PM (#35026441)

You can do even better!. You can run the second smallest one (better than GPT-3 175B) on 24GB of vram, ie LLaMA-13B. https://github.com/oobabooga/text-generation-webui/issues/14...

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by Tepix on Monday March 06, 2023 @01:45PM (#35045647)

Running it is easy but you'll probably want to finetune it, too

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by q1w2 on Friday March 03, 2023 @10:24AM (#35010789)

I wonder how many people are scrambling to set this up on their startup infra.

6x24GB NVRAM on 6 GPUs linked with NVSwitch is a little pricey, but totally doable.

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by exo-pla-net on Friday March 03, 2023 @11:58AM (#35012129)

How pricey would you estimate?

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by downvotetruth on Friday March 03, 2023 @07:30PM (#35016865)

The 11DPX-T has 11 PCIE slots (~5k https://www.abmx.com/4u-rack-server) that may function after being cut open with 11 Titans @ $200 totaling ~7k vs DGX-2 @ 400k

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by Tepix on Saturday March 04, 2023 @01:25AM (#35018709)

If you want to do it the cheap way by buying used stuff, the most expensive parts are:

- $2000 for a Threadripper 3xx5WX with a socket sWRX8 mainboard

- $5000 for 6x RTX 3090

- $350 for two 1500W PSUs

- $700 for 256GB RAM

You will also need PCIe extenders and perhaps some watercooling. And find a suitable case. The 2-card NVLink bridges are between $100 and $300 each (you nay want 3). All in all i think less than $10k.

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by touristtam on Saturday March 04, 2023 @06:06PM (#35025423)

Would rather put that in AWS.

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by Tepix on Monday March 06, 2023 @01:41PM (#35045568)

AWS is probably among the priciest options.

Privacy costs money.

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by arthurcolle on Friday March 03, 2023 @11:43PM (#35018254)

I got it running using Colab Pro+ (immediately got a V100 40GB VRAM GPU) - the 7B model works with batch size of 8 and a max seq len of 1024

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by koheripbal on Saturday March 04, 2023 @07:59AM (#35020462)

Sure, but the real value here is the 65B. Can you have multiple GPUs on colab?

[ Parent | on HN ]

Re:Facebook LLAMA is being openly distributed via torrents by arthurcolle on Saturday March 04, 2023 @11:44AM (#35022192)

I can't even get the 13B on colab to do inference with a very small sequence length.

[ Parent | on HN ]