News for hackers. Stuff that matters.
Re:Facebook LLAMA is being openly distributed via torrents by joker99 on Friday March 03, 2023 @10:38AM (#35010985)
If I may tack on a question as someone with zero clue of ML: when, if ever, will someone like me be able to run this on a Mac Studio with a M1 Ultra and 128GB of ram?
Re:Facebook LLAMA is being openly distributed via torrents by terafo on Friday March 03, 2023 @10:57AM (#35011249)
As far as I can tell you can do it right now, at least for small 13B model, not sure about bigger models.
Re:Facebook LLAMA is being openly distributed via torrents by eightysixfour on Friday March 03, 2023 @12:42PM (#35012737)
I don't believe they could, need CUDA and more VRAM...
Re:Facebook LLAMA is being openly distributed via torrents by terafo on Friday March 03, 2023 @02:07PM (#35013690)
128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.
Re:Facebook LLAMA is being openly distributed via torrents by eightysixfour on Friday March 03, 2023 @02:30PM (#35013947)
Does M1 use system memory as VRAM as well?
Re:Facebook LLAMA is being openly distributed via torrents by jml7c5 on Friday March 03, 2023 @03:16PM (#35014478)
Yes, it's all unified memory.

As much as I dislike the loss of socketed RAM, I have to say it's working out well for Apple users so far. I wonder if CXL will change the situation for consumer devices or if it will only be useful at the scale of a server rack.

Re:Facebook LLAMA is being openly distributed via torrents by MacsHeadroom on Saturday March 04, 2023 @09:25PM (#35026423)
13B fits on a single 3090 (24GB) in int8.

128 gigs might even be enough for 65B, if slowly.

Re:Facebook LLAMA is being openly distributed via torrents by TaylorAlexander on Friday March 03, 2023 @01:59PM (#35013612)
The 13B parameter model is two 13.5GB chunks, but the 7B parameter model is one 13.5GB chunk so that one might be possible.

EDIT: There seems to be a lot of overhead. Here someone struggles to fit the 7B parameter model (13.5GB checkpoint) on a 24GB card: https://github.com/facebookresearch/llama/issues/55

Re:Facebook LLAMA is being openly distributed via torrents by MacsHeadroom on Saturday March 04, 2023 @09:23PM (#35026415)
You can run 7B (equal to GPT-3 175B), 13B (better than GPT-3 175B), or 30B (better than anything else publicly available) but probably not 65B with that much RAM on an M1.

That would be using the CPU, as the M1 GPU is not yet supported.