Re:Bringing K/V context quantisation to Ollama
by sadeshmukh on Wednesday December 04, 2024 @09:25PM
(#42324186)
A lot of the UIs, including OpenWebUI have the feature to expose over LAN with users - that's what I did to use my GPU while still being on my phone. Not entirely sure about native UIs though.
Also, I normally use Groq's (with a q) API since it's really cheap with no upfront billing info required - it's a whole order of magnitude cheaper iirc than OpenAI/Claude. They literally have a /openai endpoint if you need compatibility.
You can look in the direction of Google's Gemma if you need a lightweight open weights LLM - there was something there that I forgot.
Also, I normally use Groq's (with a q) API since it's really cheap with no upfront billing info required - it's a whole order of magnitude cheaper iirc than OpenAI/Claude. They literally have a /openai endpoint if you need compatibility.
You can look in the direction of Google's Gemma if you need a lightweight open weights LLM - there was something there that I forgot.