News for hackers. Stuff that matters.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Casteil on Monday January 08, 2024 @11:25PM (#38922300)
I've had excellent results with Mixtral too - it's genuinely impressive. Only problem is that it's a relatively big model that's difficult to run with full GPU inference on consumer hardware (vs the 7b/13b models people typically use).

So far, the main consumer platform capable of running it without 'ruining' the quality of its output (with high levels of quantization) is the newer Apple Silicon Macs with unified memory - generally >=48GB. It can apparently be done on 32 or 36GB, but there's not much headroom.

Edit: As coder543 points out, yes - you can run it without more lossy levels of quantization on multi-GPU setups providing those have enough combined vram.

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by coder543 on Monday January 08, 2024 @11:41PM (#38922414)
Mixtral works great at 3-bit quantization. It fits onto a single RTX 3090 and runs at about 50 tokens/s. The output quality is not "ruined" at all.

For the amount of money you're talking about, you could also buy two 3090s (~$750 each on eBay) and have 48GB of VRAM to run with less quantization at full speed.

M-series Macs are surprisingly flexible platforms, but they're not "the only" consumer platform that can do Mixtral.

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Casteil on Monday January 08, 2024 @11:47PM (#38922440)
Fair enough. I did put 'ruining' in quotes for a reason - I haven't compared output between Q3 and Q4_K_M that I use, but you do generally sacrifice output quality at higher quantization levels.

And you're right, you can run it on a multi-GPU setup if you're so inclined.

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by coder543 on Monday January 08, 2024 @11:50PM (#38922459)
You can also choose to run at 4-bit quantization, offloading ~27 out of 33 layers to the GPU, and that runs at about 25 tokens/s for me. I think that's about the same speed as you get out of an M1 Max running at 4 bits? Although I'm not sure about the newer M2 or M3 Max chips. Googling around, I didn't immediately see clear benchmarks for those.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Casteil on Monday January 08, 2024 @11:59PM (#38922521)
Nice - that's still pretty solid.. although on a more typical 3060 or 3070 with less vram available, I probably wouldn't expect numbers quite that good.

My 14" M1 Max does around 30t/s on Mixtral Q4_K_M.

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by patrakov on Tuesday January 09, 2024 @02:26AM (#38923382)
Just as another data point, a CPU-only setup with Q5_K_M would give you roughly 4 tokens per second on a Ryzen laptop (Dell Inspiron 7415 upgraded to 64 GB of RAM).
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by eyegor on Tuesday January 09, 2024 @01:33AM (#38923092)
So you don't see significantly worse performance on 3bit quantized models compared to 4? Every 7/13b model I tried quantized gave much worse responses at 3 bit and below, whereas the differences from 4 bit to 6 or even 8 bit is more subtle.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by coder543 on Tuesday January 09, 2024 @02:04AM (#38923270)
Mixtral is a larger model, so maybe that makes it more tolerant of that level of quantization? I’ve been impressed with 3-bit Mixtral, but I haven’t done a ton of side by sides against 4-bit because I haven’t felt the need.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by 3abiton on Tuesday January 09, 2024 @03:05AM (#38923579)
Have you tried the 2x 3090 setup? Using nvlink or SLI?
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by coder543 on Tuesday January 09, 2024 @08:44AM (#38926068)
I have not personally gotten to test things that way
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Tepix on Tuesday January 09, 2024 @01:56PM (#38930308)
I tried it, with NVlink. The speedup during inference is negligible. You'll probably benefit more during training.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by kwerk on Tuesday January 09, 2024 @02:21PM (#38930595)
I use 2x 3090, no nvlink though. I’d read it doesn’t help that much but not well read on any improvement.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by chpatrick on Tuesday January 09, 2024 @04:55AM (#38924245)
Could you share what you use to run it on a single 3090? I'd love to try it!
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by coder543 on Tuesday January 09, 2024 @08:46AM (#38926079)
ollama has been by far the easiest way for me, either on Linux directly (as I do now) or WSL2.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by bekantan on Tuesday January 09, 2024 @07:04AM (#38925148)
> The output quality is not "ruined" at all.

That was my experience as well - 3-bit version is pretty good.

I also tried 2-bit version, which was disappointing.

However, there is a new 2-bit approach in the works[1] (merged yesterday) which performs surprisingly well for Mixtral 8x7B Instruct with 2.10 bits per weight (12.3 GB model size).

[1] https://github.com/ggerganov/llama.cpp/pull/4773

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by coder543 on Tuesday January 09, 2024 @10:04AM (#38926970)
That is a very interesting discussion. Weird to me that the quantization code wasn’t required to be in the same PR. Ika is also already talking about a slightly higher 2.31bpw quantization, apparently.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by mark_l_watson on Tuesday January 09, 2024 @10:16AM (#38927093)
I could only run 2-bit q2 mode on my 32G M2 Pro. I was a little disappointed, but I look forward to try the new approach you linked. I just use Mistral’s and also a 3rd party hosting service for now.

After trying the various options for running locally, I have settled on just using Ollama - really convenient and easy, and the serve APIs let me use various LLMs in several different (mostly Lisp) programming languages.

With excellent resources from Hugging Face, tool providers, etc., I hope that the user facing interface for running LLMs is simplified even further: enter your hardware specs and get available models filtered by what runs on a user’s setup. Really, we are close to being there.

Off topic: I hope I don’t sound too lazy, but I am retired (in the last 12 years before retirement I managed a deep learning team at Capital One, worked for a while at Google and three other AI companies) and I only allocate about 2 hours a day to experiment with LLMs so I like to be efficient with my time.

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Casteil on Tuesday January 09, 2024 @11:52AM (#38928377)
Ollama[1] + Ollama WebUI[2] is a killer combination for offline/fully local LLMs. Takes all the pain out of getting LLMs going. Both projects are rapidly adding functionality including recent addition of multimodal support.

[1] https://github.com/jmorganca/ollama

[2] https://github.com/ollama-webui/ollama-webui

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by weiran on Saturday January 13, 2024 @09:42AM (#38980340)
You should be able to run Q3 and maybe even Q4 quants with 32GB. Even with the GPU as you can up the max RAM allocation with: 'sudo sysctl iogpu.wired_limit_mb=12345'
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by lithiumii on Monday January 08, 2024 @11:59PM (#38922520)
Three 4060 Ti 16GB (there are single slot models) is around $1500. I think is possible to get a consumer system that's cheaper than a 48GB Mac.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Casteil on Tuesday January 09, 2024 @12:00AM (#38922526)
Yep. Edited my post to reflect as much. The MBP makes a rather nice portable package though.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by ignoramous on Monday January 08, 2024 @11:37PM (#38922391)
> the newer Apple Silicon Macs with unified memory

Mixtral has been MLXd already? Write ups, if any?

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by vessenes on Tuesday January 09, 2024 @12:29AM (#38922697)
Yes it has, actually: https://github.com/ml-explore/mlx-examples. It's right in the main repo. NB, I haven't tried this, I'm using llama.cpp with a non-K-quant quantization on my MBP.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by summarity on Tuesday January 09, 2024 @05:19AM (#38924403)
I have and don't consider MLX to be production ready. I've tested it on M1Max and M1Ultra (128) machines. It's completely non-deterministic in its resource consumption, sometimes using the GPU fully, sometimes getting seemingly stuck while processing, sometimes the GPU throttles.

However, there's one curious thing: llama.cpp _always_ leads to GPU throttling on Apple Silicon (e.g. the M1Max GPU will go from 1200MHz to around 700MHz), and then fully saturates it. In the rare cases I could get MLX to stay on the GPU, it was able to keep it at the maximum clock rate. However the unpredictable pauses and seemingly unoptimized prompt processing makes it hard to pick a winner in end-to-end tokens/s

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by LoganDark on Monday January 08, 2024 @11:42PM (#38922421)
Not to my knowledge. But because the unified memory doubles as VRAM for the onboard GPU, normal GPU acceleration can access the entire model even if it's 50+ GB. That's why ASi Macs are currently the holy grail for at-home inferencing, and also why projects like llama.cpp focus so much on ASi above all else, and why so many UIs release for macOS first before other operating systems. Certain Mac models offer up to 192GB of unified memory.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by mkesper on Tuesday January 09, 2024 @02:32AM (#38923414)
But that's not a Macbook. And a Macbook M3Max with 128GB of RAM is almost 8000€.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by zarzavat on Tuesday January 09, 2024 @08:25AM (#38925887)
Considering how inaccessible and expensive 128GB of pro-level cards is, that is believe it or not, a good price.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by LoganDark on Tuesday January 09, 2024 @06:28PM (#38933847)
Not to mention that's 128GB for a single GPU. No need to shard or split.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Terretta on Tuesday January 09, 2024 @07:14AM (#38925244)
Many options for running Mistral models in your terminal using LLM:

https://simonwillison.net/2023/Dec/18/mistral/

I liked "Using Llamafile’s OpenAI API endpoint" described there, using Justine Tunney's llamafiles for Mixtral, but the article link is out of date, as the models have been replaced with newer: https://huggingface.co/jartine

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by reexpressionist on Tuesday January 09, 2024 @09:02AM (#38926263)
Here's the direct link and I can confirm that Mixtral-8x7B-v0.1 works on M2 Ultra 128GB via MLX and is easy to setup (the longest part is just downloading the weights):

https://github.com/ml-explore/mlx-examples/tree/main/llms/mi...

We'll have a tutorial soon (next week) on combining/composing with Reexpress to add uncertainty quantification (and to use it for semantic search). A link will be here: Tutorial 6: https://re.express/guide.html

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by teilo on Tuesday January 09, 2024 @09:22AM (#38926481)
I'm running it on an M2 Max with 96GB, and have plenty of room to spare. And it's fast. Faster than I can get responses from ChatGPT.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by coder543 on Tuesday January 09, 2024 @10:09AM (#38927012)
How many tokens/s? Which quantization? If you could test Q4KM and Q3KM, it would be interesting to hear how the M2 Max does!
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by teilo on Tuesday January 09, 2024 @06:22PM (#38933788)
No quantization (8_0). The full 48GB model. As for token count, I haven't tested it on more than 200 or so.
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by pilotneko on Tuesday January 09, 2024 @07:39PM (#38934549)
Isn’t 8_0 8-bit quantization?
Re:Mixtral 8x7B: A sparse Mixture of Experts language model by teilo on Thursday January 11, 2024 @10:34AM (#38953594)
Sorry. That was a major brain fart. Yes. 8-bit quantization, and using 49G of RAM.