News for hackers. Stuff that matters.
Re:Show HN: I've built a locally running Perplexity clone by keyle on Wednesday April 03, 2024 @06:50PM (#39924346)
Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on).

I too use local 7b open-hermes and it's really good.

Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Wednesday April 03, 2024 @08:10PM (#39924989)
I'm just starting to get into downloading and testing models using llama.cpp and I'm curious which model you're actually using, since they seem to come in varying levels of quantization. Is this [0] the model page for the one you're using, or should I be looking somewhere else? What is the actual file name of the model you're using?

[0] https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GG...

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @06:54PM (#39924375)
Thanks :). It's just a lot of prompting and string parsing. There are models like "Hermes-2-Pro-Mistral" (the one from the video) which are trained to work with function signatures and outputting structured text. But at the end it's just strings in > strings out, haha. But its fun (and sometimes frustrating) to use LLMs for flow control (conditions, loops...) inside your programs.
Re:Show HN: I've built a locally running Perplexity clone by davidcollantes on Wednesday April 03, 2024 @08:08PM (#39924970)
Got a link for that one? I have found a few with Hermes-2-Mistral in the name.
Re:Show HN: I've built a locally running Perplexity clone by keyle on Wednesday April 03, 2024 @07:22PM (#39924647)
Wow, I didn't know about "Hermes 2 Pro - Mistral 7B", cheers!
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @07:35PM (#39924735)
It's my go to "structured text model" atm. Try "starling-ml-beta" (7b) for some very impressive chat capabilities. I honestly think that it outperforms GPT3 half the time.
Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Wednesday April 03, 2024 @08:12PM (#39925006)
Sorry to repeat the same question I just asked the other commenter in this thread, but could you link the model page and recommend a specific level of quantization for the models you've referenced? I'd love to play with these models and see what you're talking about.
Re:Show HN: I've built a locally running Perplexity clone by BOOSTERHIDROGEN on Wednesday April 03, 2024 @08:19PM (#39925059)
Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Wednesday April 03, 2024 @08:31PM (#39925125)
Thank you — from that page, at the bottom, I was able to find this link to what I think are the quantized versions

https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-...

If you have the time, could you explain what you mean by "Q5 is minimum"? Did you determine that by trying the different models and finding this one is best, or did someone else do that evaluation, or is that just generally accepted knowledge? Sorry, I find this whole ecosystem quite confusing still, but I'm very new and that's not your problem.

Re:Show HN: I've built a locally running Perplexity clone by BOOSTERHIDROGEN on Thursday April 04, 2024 @01:59AM (#39927029)
It's the best balance if you have limited compute performance.
Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Thursday April 04, 2024 @12:51PM (#39932897)
Thank you
Re:Show HN: I've built a locally running Perplexity clone by d-z-m on Thursday April 04, 2024 @05:13AM (#39928129)
Talking GGUF, Usually the higher you can afford to go wrt. quantization(e.g. Q5 is better than Q4, etc), the better. A Q6_K has minimal performance loss from the Q8, so in most cases if you can fit a Q6_K it's recommended to just use that. TheBloke's READMEs[0] usually have a good table summarizing each quantization level.

If you're RAM constrained, you'll also have to make trade-offs about the context length. e.g. you could have 8 GB RAM and a Q5 quant with shorter context, vs Q3 with longer, etc.

[0]:https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF

Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Thursday April 04, 2024 @12:51PM (#39932899)
Thank you!
Re:Show HN: I've built a locally running Perplexity clone by madacol on Friday April 05, 2024 @09:31AM (#39942149)
Have you considered using grammar sampling?
Re:Show HN: I've built a locally running Perplexity clone by viksit on Wednesday April 03, 2024 @07:08PM (#39924495)
curious what hardware you use? and is any of this runnable on an m1 laptop?
Re:Show HN: I've built a locally running Perplexity clone by keyle on Wednesday April 03, 2024 @07:11PM (#39924525)
Absolutely, 7B will run comfortably on 16GB of RAM and most consumer level hardware. Some of the 40B run on 32GB, but it depends on the model I found (GGUF, crossing fingers help).

I ran this originally on a M1 with 32GB, I run this on an Air M2 with 16GB (and mac mini M2 32GB), no problem.

I use llama.cpp with a SwiftUI interface (my own), all native, no scripts python/js/web.

7b is obviously less capable but the instant response makes it worth exploring. It's very useful as a Google search replacement that is instantly more valuable, for general questions, than dealing with the hellscape of blog spam ruling Google atm.

Note, for my complex code queries at $dayjob where time is of the essence, I still use GPT4 plus, which is still unmatched imho, without running special hardware at least.

Re:Show HN: I've built a locally running Perplexity clone by regularfry on Thursday April 04, 2024 @06:06AM (#39928449)
I've been occasionally using a 7b Q4 quant on llama.cpp on an 8GB M1. It's usable, if not amazing.
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @07:33PM (#39924720)
Depends on your m1 specs, but should definitely be able to run a 7b model (at least with some quantization).
Re:Show HN: I've built a locally running Perplexity clone by windexh8er on Wednesday April 03, 2024 @08:59PM (#39925306)
Have you looked into tools like CrewAI [0]?

[0] https://www.crewai.io/