News for hackers. Stuff that matters.
Re:ChatGPT for Teams by eurekin on Thursday January 11, 2024 @05:32AM (#38950352)
How are you accessing it?
Re:ChatGPT for Teams by gexla on Thursday January 11, 2024 @05:47AM (#38950465)
You can access it through Poe.com.
Re:ChatGPT for Teams by wruza on Thursday January 11, 2024 @08:43AM (#38952057)
Local setup, “text generation webui”, TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF (Q4_K_M) on HF. You can run it on a decent intel cpu, takes around 32.5GB of ram including os (8gb for me). GPU with tensorcores can speed up few layers if you have one, but isn’t required. I get around 2.5-3 t/s with 8700 and 4070ti, that’s enough for chats that require some thinking.

Edit: I was using 2k window, a larger one would probably eat more ram. But even with 2k it didn’t feel like it loses context or something.

Re:ChatGPT for Teams by lhl on Thursday January 11, 2024 @01:14PM (#38956221)
For macOS and Linux, Ollama is probably the easiest way to try Mixtral (and a large number of models) locally. LM Studio is also nice and available for Mac, Windows, and Linux.

As these models can be quite large and memory intensive, if you want to just give it a quick spin, huggingface.co/chat, chat.nbox.ai, and labs.pplx.ai all have Mixtral hosted atm.