Re:Mistral AI Launches New 8x22B MOE Model
by varunvummadi on Tuesday April 09, 2024 @10:44PM
(#39986471)
The easiest is to use vllm (https://github.com/vllm-project/vllm) to run it on a Couple of A100's, and you can benchmark this using this library (https://github.com/EleutherAI/lm-evaluation-harness)
Re:Mistral AI Launches New 8x22B MOE Model
by sheepscreek on Wednesday April 10, 2024 @06:21PM
(#39996309)
In that regard, it’s even easier to use one Apple Studio with sufficient RAM and llama.cpp or even PyTorch for inference.
Re:Mistral AI Launches New 8x22B MOE Model
by fbdab103 on Tuesday April 09, 2024 @10:44PM
(#39986475)
I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format).
You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile?
Re:Mistral AI Launches New 8x22B MOE Model
by noman-land on Tuesday April 09, 2024 @10:52PM
(#39986513)
+1 on llamafile. You can point it to a custom model.
llamafile author here. I'm downloading Mixtral 8x22b right now. I can't say for certain it'll work until I try it, but let's keep our fingers crossed! If not, we'll be shipping a release as soon as possible that gets it working.
My recent work optimizing CPU evaluation https://justine.lol/matmul/ may have come at just the right time. Mixtral 8x7b always worked best at Q5_K_M and higher, which is 31GB. So unless you've got 4x GeForce RTX 4090's in your computer, CPU inference is going to be the best chance you've got at running 8x22b at top fidelity.
Re:Mistral AI Launches New 8x22B MOE Model
by moffkalast on Wednesday April 10, 2024 @07:16AM
(#39989383)
Correct me if I'm wrong, but in the tests I've run, the matmul optimizations only have an effect if there's no other blas acceleration. If one can at least offload the KV cache to cublas or run with openblas it's not really used, right? At least I didn't see any speedup in with that config when comparing that PR to the main llama.cpp branch.
The code that launches my code (see ggml_compute_forward_mul_mat) comes after CLBLAST, Accelerate, and OpenBLAS. The latter take precedence. So if you're not seeing any speedup in enabling them, it's probably because tinyBLAS has reached terms of equality with the BLAS. It's obviously nowhere near as fast as cuBLAS, but maybe PCIE memory transfer overhead explains it.
Re:Mistral AI Launches New 8x22B MOE Model
by mritchie712 on Wednesday April 10, 2024 @10:08AM
(#39990919)
you can try it on together here:
https://api.together.xyz/playground/language/mistralai/Mixtr...
Re:Mistral AI Launches New 8x22B MOE Model
by hmottestad on Wednesday April 10, 2024 @12:14PM
(#39992389)
LM Studio is a great way to test out LLMs on my MacBook: https://lmstudio.ai/
Really easy to search huggingface for new models to test directly in the app.
Re:Mistral AI Launches New 8x22B MOE Model
by LeoPanthera on Wednesday April 10, 2024 @03:34PM
(#39994869)
Make sure you get the prompt template set correctly, the defaults are wrong for a lot of models.
Re:Mistral AI Launches New 8x22B MOE Model
by unifer1 on Wednesday April 10, 2024 @07:50PM
(#39996907)
Could you explain how to do this properly ? I've been having problems with the app and am wondering if this is ehy
Re:Mistral AI Launches New 8x22B MOE Model
by LeoPanthera on Wednesday April 10, 2024 @08:23PM
(#39997158)
Look at the HuggingFace page for the model you are using. (The original page, not the page for the GGUF conversion, if necessary.) This will explain the chat format you need to use.
Re:Mistral AI Launches New 8x22B MOE Model
by bevekspldnw on Thursday April 11, 2024 @11:14AM
(#40003079)
There is a user called The Bloke on hugging face- they release pre quantized models pretty soon after the full size drop. Just watch their page and pray you can fit the 4 bit in your GPU.
I’m sure they are already working on it.
Re:Mistral AI Launches New 8x22B MOE Model
by nathanasmith on Thursday April 11, 2024 @12:54PM
(#40004287)
TheBloke stopped uploading in January. There are others that have stepped up though.
Re:Mistral AI Launches New 8x22B MOE Model
by bevekspldnw on Thursday April 11, 2024 @02:20PM
(#40005175)
Oh really? Who else should I be looking at?
That person is a hero, super bummed!
TheBloke's grant ran out.
Re:Mistral AI Launches New 8x22B MOE Model
by MPSimmons on Thursday April 11, 2024 @01:18PM
(#40004541)
I think 4b for this is support to be over 70GB, so definitely still heavy hardware.
Re:Mistral AI Launches New 8x22B MOE Model
by bevekspldnw on Thursday April 11, 2024 @02:40PM
(#40005366)
Fucking hell, my A6000 is shy of that and I can’t reasonably justify picking up a second.
Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?