Re:I Run LLMs Locally
by noman-land on Sunday December 29, 2024 @03:58PM
(#42543036)
Re:I Run LLMs Locally
by throwaway314155 on Sunday December 29, 2024 @04:20PM
(#42543262)
I love what llamafile is doing, but I'm primarily interested in a frontend for ollama, as I prefer their method of model/weights distribution. Unless I'm wrong, llamafile serves as both the frontend and backend.
Re:I Run LLMs Locally
by DrBenCarson on Sunday December 29, 2024 @05:13PM
(#42543849)
LM Studio?
If I understand the distinction correctly, I run llamafile as a backend. I start it with the filename of a model on the command-line (might need a -M flag or something) and it will start up a chat-prompt for interaction in the terminal but also opens a port that speaks some protocol that I can connect to using a frontend (in my case usually gptel in emacs).
I ran it temporarily in Docker the other day. It was a 8gb image. I'm unsure why a webui is 8gb.
Some of the features (RAG retrieval) now use embeddings that are calculated in Open WebUI rather than in Ollama or another backend. It does seem like it'd be nice for them to refactor to make things like that optional for those who want a simpler interface, but then again, there are plenty of other lighter-weight options.
Given how much data (personal and otherwise) that you're likely going to feeding into it, I would HIGHLY recommend using an open-source chat interface like Jan.
It's cross platform, works with most LLMs, and has extension support.
Another alternative is LibreChat, it's more fully featured but its heavier weight and spins up quite a few docker instances.
https://github.com/danny-avila/LibreChat
If you're going through the trouble of running models locally, it doesn't make sense to couple that with a proprietary closed source interface (like LM Studio).
Re:I Run LLMs Locally
by synapsomorphy on Sunday December 29, 2024 @07:09PM
(#42544884)
Did you misread? No one mentioned LM studio, open-webui (frontend) is open source and usually uses ollama (backend) which is also open source.
Jan and Librechat are both less popular than either open-webui or ollama, how do their features compare?
Re:I Run LLMs Locally
by throwaway314155 on Sunday December 29, 2024 @07:30PM
(#42545040)
No one had suggested LM Studio at this point in the thread.
Do you really care? Open WebUI has excellent UI and features, and its complexity is well hidden, IMO.
Re:I Run LLMs Locally
by throwaway314155 on Sunday December 29, 2024 @06:51PM
(#42544737)
I agree it's a fine interface, clearly a lot of work has gone into making it polished and feature-rich. I'm simply seeking alternatives. If there aren't any with the same feature set, I'll probably just use Open WebUI.
Off the topic: I am also in favor of minimalist.
In the case of AlphaGeometry, I made AlphaGeometryRE to get rid of tensorflow/jax/flax, and 100+ Python packages.
Check out https://recurse.chat/
Does anyone know of a lighter/minimalist version?