News for hackers. Stuff that matters.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by LoganDark on Monday January 08, 2024 @11:42PM (#38922421)
Not to my knowledge. But because the unified memory doubles as VRAM for the onboard GPU, normal GPU acceleration can access the entire model even if it's 50+ GB. That's why ASi Macs are currently the holy grail for at-home inferencing, and also why projects like llama.cpp focus so much on ASi above all else, and why so many UIs release for macOS first before other operating systems. Certain Mac models offer up to 192GB of unified memory.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by mkesper on Tuesday January 09, 2024 @02:32AM (#38923414)
But that's not a Macbook. And a Macbook M3Max with 128GB of RAM is almost 8000€.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by zarzavat on Tuesday January 09, 2024 @08:25AM (#38925887)
Considering how inaccessible and expensive 128GB of pro-level cards is, that is believe it or not, a good price.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by LoganDark on Tuesday January 09, 2024 @06:28PM (#38933847)
Not to mention that's 128GB for a single GPU. No need to shard or split.