News for hackers. Stuff that matters.
Re:Show HN: I made an app to use local AI as daily driver by pentagrama on Tuesday February 27, 2024 @10:11PM (#39533494)
Sadly I can't try this because I'm on Windows or Linux.

Was testing apps like this if anyone is interested:

Best / Easy to use:

- https://lmstudio.ai

- https://msty.app

- https://jan.ai

More complex / Unpolished UI:

- https://gpt4all.io

- https://pinokio.computer

- https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generat...

- https://github.com/LostRuins/koboldcpp

Misc:

- https://faraday.dev (AI Characters):

No UI / Command line (not for me):

- https://ollama.com

- https://privategpt.dev

- https://serge.chat

- https://github.com/Mozilla-Ocho/llamafile

Pending to check:

- https://recurse.chat

Feel free to recommend more!

Re:Show HN: I made an app to use local AI as daily driver by chown on Tuesday February 27, 2024 @10:27PM (#39533597)
I am the author of Msty app mentioned here. So humbled to see an app that is just about a month old that I mostly wrote for my wife and some friends to begin with (who got overwhelmed with everything that was going in LLM world), on the top of your list. Thank you!
Re:Show HN: I made an app to use local AI as daily driver by Datagenerator on Wednesday February 28, 2024 @01:54AM (#39534866)
Looks interesting, but can't see what it is doing. Any link to the source code?
Re:Show HN: I made an app to use local AI as daily driver by petemir on Wednesday February 28, 2024 @02:20AM (#39535008)
If you need help for testing the Linux version let me know, I’d be happy to help
Re:Show HN: I made an app to use local AI as daily driver by chown on Wednesday February 28, 2024 @05:05AM (#39536033)
I was actually looking for one! What's the best way to reach you? Mind jumping on our Discord so that I can share the installer with you soon?
Re:Show HN: I made an app to use local AI as daily driver by crooked-v on Wednesday February 28, 2024 @03:15AM (#39535357)
One bit of feedback: there's nowhere to put system messages. These can be much more influential than user prompts when it comes to shaping the tone and style of the response.
Re:Show HN: I made an app to use local AI as daily driver by chown on Wednesday February 28, 2024 @05:09AM (#39536060)
That's on the top of our list. It got pushed back because we want to support creating a character/profile (basically select a model and apply some defaults including a system prompt). But I feel like that was a mistake tomwait for it. Regardless, it is getting added in the next release (the one after something that is dropping in a day or 2, which is a big release in itself)
Re:Show HN: I made an app to use local AI as daily driver by hanniabu on Wednesday February 28, 2024 @04:04PM (#39543401)
1) What are the mac system requirements? Does it need a specific OS version?

2) If you're privacy first, many would feel a lot more comfortable if this was released as an app in the app store so it will be sandboxed. This is important because it's not open source so we have no idea what is happening in the background. Alternatively open source it, which many here have requested.

Re:Show HN: I made an app to use local AI as daily driver by visarga on Wednesday February 28, 2024 @12:17AM (#39534287)
Add Open-WebUI (used to be Ollama-WebUI)

https://github.com/open-webui/open-webui

a well featured UI with very active team

Re:Show HN: I made an app to use local AI as daily driver by lolpanda on Tuesday February 27, 2024 @11:26PM (#39533985)
Oh thanks! didn't know there are quite a few ChatGPT local alternatives. I was wondering what users they are targeting. Engineers or average users? I guess average users will likely choose ChatGPT and Perplexity over local apps for more recent knowledge of the world.
Re:Show HN: I made an app to use local AI as daily driver by chown on Tuesday February 27, 2024 @11:46PM (#39534118)
Hi. I'm the author of Msty app, 2nd on the list above. You are right about average users likely choosing ChatGPT over local models. My wife was the first and the biggest user of my app. A software engineer by profession and training but she likes to not worry about LLM world and just to use it as a tool that makes you more productive. As soon as she took Msty for a ride, I realized that some users, despite their background, care about online models. This actually led me adding support for online models right away. However, she really likes to make use of the parallel chat feature and uses both Mistral and ChatGPT models to give same prompt and then compare the output and choose the best answer (or sometimes make a hybrid choice). She says that being able to compare multiple outputs like that is a tremendously helpful. But that's the extent of local LLMs for her. So far my effort has been to target a bit higher than the average users while making it approachable for more advanced users as well.
Re:Show HN: I made an app to use local AI as daily driver by Gunnerhead on Tuesday February 27, 2024 @11:57PM (#39534170)
I’m looking for a ChatGPT client alternative, i.e. I can use my own OpenAI API key in some other client.

Offline isn’t important for me, only that $20 is a lot of money, when I’d wager most months my usage is a lot less. However, I’d still want access to completion, DALL-E, etc.

Would Msty be a good option for me?

Re:Show HN: I made an app to use local AI as daily driver by chown on Wednesday February 28, 2024 @12:05AM (#39534211)
Give it a try and see how you feel. "Yes, it will" be a dishonest answer to be completely honest at least at this point. The app has been out for just about a month and I am still working in it. I would love a user like you to give it a try and give me some feedback (please). I am very active on our Discord if you want to get in touch (just mention your HN username and I will wace).
Re:Show HN: I made an app to use local AI as daily driver by Gunnerhead on Wednesday February 28, 2024 @12:49AM (#39534501)
Thank you so much, I’m excited to give this a try in the next few days.
Re:Show HN: I made an app to use local AI as daily driver by AriedK on Wednesday February 28, 2024 @03:14AM (#39535345)
Looks great, though the fact that you have to ignore your anti-virus warning during installation, and the fact that it phones home (to insights.msty.app) directly after launch despite the line in the FAQ on not collecting any data makes me a little skittish.
Re:Show HN: I made an app to use local AI as daily driver by joshmarinacci on Tuesday February 27, 2024 @11:43PM (#39534105)
Do any of these let you dump in a bunch of your own documents to use as a corpus and then query and summarize them ?
Re:Show HN: I made an app to use local AI as daily driver by chown on Wednesday February 28, 2024 @12:08AM (#39534223)
Author of Msty here. Not yet but I am already working on the design for it to be added in very near future. I am happy to chat more with you to understand your needs and what you are looking in such apps. Please hop on the Discord if you don't mind :)
Re:Show HN: I made an app to use local AI as daily driver by hanniabu on Wednesday February 28, 2024 @03:59PM (#39543344)
Some of my usecases would be summarizing a PDF report, analyzing json/csv data, upload a dev project to write a function or feature or build a UI, rename image files, categorize images, etc
Re:Show HN: I made an app to use local AI as daily driver by windexh8er on Tuesday February 27, 2024 @11:47PM (#39534122)
Yes, GPT4All has RAG-like features. Basically you configure some directories and then have it load docs from whatever folders you have enabled for the model you're currently using. I haven't used it a ton, but I have used it to review long documents and it's worked well depending on the model.
Re:Show HN: I made an app to use local AI as daily driver by 8n4vidtmkvmk on Wednesday February 28, 2024 @12:47AM (#39534492)
The new one straight from Nvidia does I believe.
Re:Show HN: I made an app to use local AI as daily driver by Datagenerator on Wednesday February 28, 2024 @01:58AM (#39534883)
Open-WebUI has support for doing that, it works using #tags for each document so you can ask questions about multiple specific documents.
Re:Show HN: I made an app to use local AI as daily driver by greggsy on Wednesday February 28, 2024 @03:11AM (#39535320)
Re:Show HN: I made an app to use local AI as daily driver by stlhood on Tuesday February 27, 2024 @11:33PM (#39534045)
Just FYI, llamafile includes a web-based chat UI. It fires up automatically.
Re:Show HN: I made an app to use local AI as daily driver by theolivenbaum on Wednesday February 28, 2024 @12:52AM (#39534527)
We just added local LLM support to our curiosity.ai app too - if anyone wants to try we're looking for feedback there!
Re:Show HN: I made an app to use local AI as daily driver by hmdai on Wednesday February 28, 2024 @02:47AM (#39535189)
Try this one: https://uneven-macaw-bef2.hiku.app/app/

It loads the LLM in the browser, using webgpu, so it works offline after the first load, it's also PWA you can install. It should work on chrome > 113 on desktop and chrome > 121 on mobile.

Re:Show HN: I made an app to use local AI as daily driver by wanderingmind on Wednesday February 28, 2024 @02:59AM (#39535258)
lmstudio is using a dark pattern I really hate. Don't have a Github logo in your webpage if your software is not source available. It just takes to Github to some random config repos they have. This is poor choice in my opinion.
Re:Show HN: I made an app to use local AI as daily driver by Hugsun on Wednesday February 28, 2024 @07:04AM (#39536809)
We call that stolen valor.
Re:Show HN: I made an app to use local AI as daily driver by woadwarrior01 on Wednesday February 28, 2024 @03:02AM (#39535272)
Since I couldn't find it in your list, I'd like to plug my own macOS (and iOS) app: Private LLM. Unlike almost every other app in the space, it isn't based on llama.cpp (we use mlc-llm) or naive RTN quantized models (we use OmniQuant). Also, the app has deep integrations with macOS and iOS (Shortcuts, Siri, macOS Services, etc).

Incidentally, it currently runs Mixtral 8x7B Instruct[2] and Mistral[3] models faster than any other macOS app. The comparison videos are with Ollama, but it generalizes well to almost every other macOS app that I've seen uses llama.cpp for inference. :)

nb: Mixtral 8x7B Instruct requires an Apple Silicon Mac with at least 32GB of RAM.

[1]: https://privatellm.app/

[2]: https://www.youtube.com/watch?v=CdbxM3rkxtc

[3]: https://www.youtube.com/watch?v=UIKOjE9NJU4

Re:Show HN: I made an app to use local AI as daily driver by iknowstuff on Wednesday February 28, 2024 @04:14AM (#39535717)
MacGPT is way handy because of a global keyboard shortcut which opens a spotlight-like prompt. I would love to have a local equivalent
Re:Show HN: I made an app to use local AI as daily driver by sigmoid10 on Wednesday February 28, 2024 @03:17AM (#39535373)
What's the performance like in tokens/s?
Re:Show HN: I made an app to use local AI as daily driver by woadwarrior01 on Wednesday February 28, 2024 @04:26AM (#39535780)
You can see ms/token in a tiny font on the top of the screen, once the text generation completes in both the videos I'd linked to. Performance will vary by machine. On my 64GB M2 Mac Studio Max, I get ~47 tokens/s (21.06ms/token) with Mistral Instruct v0.2 and ~33 tokens/s (30.14ms/token) with Mixtral Instruct v0.1.
Re:Show HN: I made an app to use local AI as daily driver by castles on Wednesday February 28, 2024 @07:22AM (#39536964)
Interesting! What's the prompt eval processing speed like compared to llama.cpp and kin?
Re:Show HN: I made an app to use local AI as daily driver by woadwarrior01 on Wednesday February 28, 2024 @07:51AM (#39537195)
I haven't run any specific low level benchmarks, lately. But chunked prefilling and tvm auto-tuned Metal kernels from mlc-llm seemed to make a big differenced, the last time I checked. Also, compared to stock mlc-llm, I use a newer version of metal (3.0) and have a few modifications to make models have a slightly smaller memory and disk footprint, also slightly faster execution. Because unlike the mlc-llm folks, I only care about compatibility with Apple platforms. They support so much more than that in their upstream project.
Re:Show HN: I made an app to use local AI as daily driver by castles on Wednesday February 28, 2024 @08:34AM (#39537668)
thanks, I'll give it a crack
Re:Show HN: I made an app to use local AI as daily driver by vorticalbox on Wednesday February 28, 2024 @03:41AM (#39535530)
have you seen llamafile[0]?

[0] https://github.com/Mozilla-Ocho/llamafile

Re:Show HN: I made an app to use local AI as daily driver by greggsy on Wednesday February 28, 2024 @04:45AM (#39535898)
Khoj was one of the first 'low-touch' solutions out there I think. It's ok, but still under active development, like all of them really.

https://khoj.dev/

Re:Show HN: I made an app to use local AI as daily driver by quickthrower2 on Wednesday February 28, 2024 @05:33AM (#39536261)
Thanks for the list. Tried Jan just now as it is both easy and open source. It is a bit buggy I think but the concept is ace. The quick install, tells you which models work on your machine, one click download and then a chatgpt style interface. Mistral 7B running on my low spec laptop at 6 token/s making some damn sense is amazing. The bugs are at the inference time. Could be hardware issues though, not sure. YMMV
Re:Show HN: I made an app to use local AI as daily driver by chaxor on Wednesday February 28, 2024 @11:06AM (#39539785)
What about https://github.com/open-webui/open-webui ?

Seems to have more features than all of them