News for hackers. Stuff that matters.
Re:Facebook LLAMA is being openly distributed via torrents by terafo on Friday March 03, 2023 @10:57AM (#35011249)
As far as I can tell you can do it right now, at least for small 13B model, not sure about bigger models.
Re:Facebook LLAMA is being openly distributed via torrents by eightysixfour on Friday March 03, 2023 @12:42PM (#35012737)
I don't believe they could, need CUDA and more VRAM...
Re:Facebook LLAMA is being openly distributed via torrents by terafo on Friday March 03, 2023 @02:07PM (#35013690)
128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.
Re:Facebook LLAMA is being openly distributed via torrents by eightysixfour on Friday March 03, 2023 @02:30PM (#35013947)
Does M1 use system memory as VRAM as well?
Re:Facebook LLAMA is being openly distributed via torrents by jml7c5 on Friday March 03, 2023 @03:16PM (#35014478)
Yes, it's all unified memory.

As much as I dislike the loss of socketed RAM, I have to say it's working out well for Apple users so far. I wonder if CXL will change the situation for consumer devices or if it will only be useful at the scale of a server rack.

Re:Facebook LLAMA is being openly distributed via torrents by MacsHeadroom on Saturday March 04, 2023 @09:25PM (#35026423)
13B fits on a single 3090 (24GB) in int8.

128 gigs might even be enough for 65B, if slowly.

Re:Facebook LLAMA is being openly distributed via torrents by TaylorAlexander on Friday March 03, 2023 @01:59PM (#35013612)
The 13B parameter model is two 13.5GB chunks, but the 7B parameter model is one 13.5GB chunk so that one might be possible.

EDIT: There seems to be a lot of overhead. Here someone struggles to fit the 7B parameter model (13.5GB checkpoint) on a 24GB card: https://github.com/facebookresearch/llama/issues/55