Re:Mixtral 8x7B: A sparse Mixture of Experts language model
by LoganDark on Monday January 08, 2024 @11:42PM
(#38922421)
Not to my knowledge. But because the unified memory doubles as VRAM for the onboard GPU, normal GPU acceleration can access the entire model even if it's 50+ GB. That's why ASi Macs are currently the holy grail for at-home inferencing, and also why projects like llama.cpp focus so much on ASi above all else, and why so many UIs release for macOS first before other operating systems. Certain Mac models offer up to 192GB of unified memory.
[ Parent
| on HN ]
Re:Mixtral 8x7B: A sparse Mixture of Experts language model
by mkesper on Tuesday January 09, 2024 @02:32AM
(#38923414)
But that's not a Macbook. And a Macbook M3Max with 128GB of RAM is almost 8000€.