News for hackers. Stuff that matters.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Casteil on Monday January 08, 2024 @11:47PM (#38922440)
Fair enough. I did put 'ruining' in quotes for a reason - I haven't compared output between Q3 and Q4_K_M that I use, but you do generally sacrifice output quality at higher quantization levels.

And you're right, you can run it on a multi-GPU setup if you're so inclined.

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by coder543 on Monday January 08, 2024 @11:50PM (#38922459)
You can also choose to run at 4-bit quantization, offloading ~27 out of 33 layers to the GPU, and that runs at about 25 tokens/s for me. I think that's about the same speed as you get out of an M1 Max running at 4 bits? Although I'm not sure about the newer M2 or M3 Max chips. Googling around, I didn't immediately see clear benchmarks for those.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Casteil on Monday January 08, 2024 @11:59PM (#38922521)
Nice - that's still pretty solid.. although on a more typical 3060 or 3070 with less vram available, I probably wouldn't expect numbers quite that good.

My 14" M1 Max does around 30t/s on Mixtral Q4_K_M.

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by patrakov on Tuesday January 09, 2024 @02:26AM (#38923382)
Just as another data point, a CPU-only setup with Q5_K_M would give you roughly 4 tokens per second on a Ryzen laptop (Dell Inspiron 7415 upgraded to 64 GB of RAM).