Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data...
by kwerk on Wednesday March 27, 2024 @10:21AM
(#39839526)
I have two 3090s and it runs fine with `ollama run mixtral`. Although OP definitely meant mistral with the 7B note
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data...
by jsight on Wednesday March 27, 2024 @10:47AM
(#39839925)
ollama run mixtral will default to the quantized version (4bit IIRC). I'd guess this is why it can fit with two 3090s.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data...
by Havoc on Wednesday March 27, 2024 @12:20PM
(#39841204)
The smaller quants still require a 24gb card. 16 might work but doubt it
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data...
by XCSme on Wednesday March 27, 2024 @11:06AM
(#39840250)
I have 128GB, but something is weird with Ollama. Even though for the Ollama Docker I only allow 90GB, it ends up using 128GB/128GB, so the system because very slow (mouse freezes).
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data...
by InitEnabler on Wednesday March 27, 2024 @04:34PM
(#39844257)
What docker flags are you running?
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data...
by XCSme on Wednesday March 27, 2024 @07:30PM
(#39845903)
None? The default ones from their docs.
The Docker also shows minimal usage for the ollama server which is also strange.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data...
by XCSme on Wednesday March 27, 2024 @11:02AM
(#39840192)
Sorry, it was from memory.
I have those models in Ollama:
I have those:
dolphin-mixtral:latest (24.6GB) mistral:latest (3.8GB)
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data...
by ranger_danger on Wednesday March 27, 2024 @10:27AM
(#39839598)
I'm using mixtral-8x7b-v0.1.Q4_K_M.gguf with llama.cpp and it only requires 25GB.
Do you mean mistral?
mixtral is 8x7B and requires like 100GB of RAM
Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version