News for hackers. Stuff that matters.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by jasonjmcghee on Wednesday March 27, 2024 @10:19AM (#39839491)
> mixtral works well

Do you mean mistral?

mixtral is 8x7B and requires like 100GB of RAM

Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by kwerk on Wednesday March 27, 2024 @10:21AM (#39839526)
I have two 3090s and it runs fine with `ollama run mixtral`. Although OP definitely meant mistral with the 7B note
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by jsight on Wednesday March 27, 2024 @10:47AM (#39839925)
ollama run mixtral will default to the quantized version (4bit IIRC). I'd guess this is why it can fit with two 3090s.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by Havoc on Wednesday March 27, 2024 @12:20PM (#39841204)
The smaller quants still require a 24gb card. 16 might work but doubt it
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Wednesday March 27, 2024 @11:06AM (#39840250)
I have 128GB, but something is weird with Ollama. Even though for the Ollama Docker I only allow 90GB, it ends up using 128GB/128GB, so the system because very slow (mouse freezes).
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by InitEnabler on Wednesday March 27, 2024 @04:34PM (#39844257)
What docker flags are you running?
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Wednesday March 27, 2024 @07:30PM (#39845903)
None? The default ones from their docs.

The Docker also shows minimal usage for the ollama server which is also strange.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Wednesday March 27, 2024 @11:02AM (#39840192)
Sorry, it was from memory.

I have those models in Ollama:

I have those:

dolphin-mixtral:latest (24.6GB) mistral:latest (3.8GB)

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by ranger_danger on Wednesday March 27, 2024 @10:27AM (#39839598)
I'm using mixtral-8x7b-v0.1.Q4_K_M.gguf with llama.cpp and it only requires 25GB.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by chpatrick on Wednesday March 27, 2024 @10:39AM (#39839797)
The quantized one works fine on my 24GB 3090.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by K0balt on Wednesday March 27, 2024 @10:26AM (#39839589)
I run mixtral 6 bit quant very happily on my MacBook with 64 gb.