Mixtral 8x7B: A sparse Mixture of Experts language model

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by aunty_helen on Tuesday January 09, 2024 @12:06AM (#38922564)

On mac silicon:

https://ollama.ai/

ollama pull mixtral

For a chatgpt-esk web ui

https://github.com/ollama-webui/ollama-webui

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v ollama-webui:/app/backend/data --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main

Navigate to http://localhost:3000

You can also use ollama in langchain.

[ Parent | on HN ]

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by viraptor on Tuesday January 09, 2024 @12:22AM (#38922655)

There's also some unlocked fine tunings available. Dolphin seems to be a very popular one. (Trained on more coding data) If you want to fit under 32gb, there's https://ollama.ai/library/dolphin-mixtral:8x7b-v2.7-q3_K_M

[ Parent | on HN ]

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by NwpierratorR on Tuesday January 09, 2024 @07:05AM (#38925157)

I haven't played with dolphin-mixtral, but dolphin-mistral gave me a good impression for generic rag application.

There's also developments and experimentation in making it more factual via dpo and laser(afaik so far not very unsuccessful).

[ Parent | on HN ]

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by vunderba on Tuesday January 09, 2024 @03:01AM (#38923567)

I wouldn't use Ollama (or any LLM host) through docker on Mac m1 since as far as I'm aware there's still no support for Metal.

[ Parent | on HN ]

Re:Mixtral 8x7B: A sparse Mixture of Experts language model by Luc on Tuesday January 09, 2024 @04:21AM (#38923989)

The UI is in Docker but the Ollama server isn't.

[ Parent | on HN ]