Re:Facebook LLAMA is being openly distributed via torrents
by terafo on Friday March 03, 2023 @10:57AM
(#35011249)
As far as I can tell you can do it right now, at least for small 13B model, not sure about bigger models.
[ Parent
| on HN ]
Re:Facebook LLAMA is being openly distributed via torrents
by eightysixfour on Friday March 03, 2023 @12:42PM
(#35012737)
I don't believe they could, need CUDA and more VRAM...
Re:Facebook LLAMA is being openly distributed via torrents
by terafo on Friday March 03, 2023 @02:07PM
(#35013690)
128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.
Re:Facebook LLAMA is being openly distributed via torrents
by eightysixfour on Friday March 03, 2023 @02:30PM
(#35013947)
Does M1 use system memory as VRAM as well?
Re:Facebook LLAMA is being openly distributed via torrents
by jml7c5 on Friday March 03, 2023 @03:16PM
(#35014478)
Yes, it's all unified memory.
As much as I dislike the loss of socketed RAM, I have to say it's working out well for Apple users so far. I wonder if CXL will change the situation for consumer devices or if it will only be useful at the scale of a server rack.
Re:Facebook LLAMA is being openly distributed via torrents
by MacsHeadroom on Saturday March 04, 2023 @09:25PM
(#35026423)
13B fits on a single 3090 (24GB) in int8.
128 gigs might even be enough for 65B, if slowly.
Re:Facebook LLAMA is being openly distributed via torrents
by TaylorAlexander on Friday March 03, 2023 @01:59PM
(#35013612)
The 13B parameter model is two 13.5GB chunks, but the 7B parameter model is one 13.5GB chunk so that one might be possible.
EDIT: There seems to be a lot of overhead. Here someone struggles to fit the 7B parameter model (13.5GB checkpoint) on a 24GB card: https://github.com/facebookresearch/llama/issues/55