News for hackers. Stuff that matters.
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @06:52PM (#39924357)
Happy to answer any questions and open for suggestions :)

It's basically a LLMs with access to a search engine and the ability to query a vector db.

The top n results from each search query (initialized by the LLM) will be scraped, split into little chunks and saved to the vector db. The LLM can then query this vector db to get the relevant chunks. This obviously isn't as comprehensive as having a 128k context LLM just summarize everything, but at least on local hardware it's a lot faster and way more resource friendly. The demo on GitHub runs on a normal consumer GPU (amd rx 6700xt) with 12gb vRAM.

Re:Show HN: I've built a locally running Perplexity clone by koeng on Wednesday April 03, 2024 @08:14PM (#39925018)
What is the search engine that it uses?
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @08:19PM (#39925056)
searxng, which is a locally running meta search engine combining a lot of different sources (including Google and co)
Re:Show HN: I've built a locally running Perplexity clone by mmahemoff on Thursday April 04, 2024 @01:27AM (#39926852)
This might be more of a searxng question, but doesn't it quickly run up against anti-bot measures? CAPTCHA challenges and Forbidden responses? I can see the manual has some support for dealing with CAPTCHA [1], but in practical terms, I would guess a tool like this can't be used extensively all day long.

I'm wondering if there's a search API that would make the backend seamless for something like this.

1. https://docs.searxng.org/admin/answer-captcha.html

Re:Show HN: I've built a locally running Perplexity clone by visarga on Thursday April 04, 2024 @02:24AM (#39927147)
As a last resort we could have AI work on top of a real web browser and solving captchas as well. Should look like normal usage. I think these kinds of systems LLM + RAG + Web Agent will become widespread and the preferred method to interact with the web.

We can escape all ads and dark UI patterns by delegating this task to AI agents. We could have it collect our feeds, filter, rank and summarize them to our preferences, not theirs. I think every web browser, operating system and mobile device will come equipped with its own LLM agent.

The development of AI screen agents will probably get a big boost from training on millions of screen capture videos with commentary on YouTube. They will become a major point of competition on features. Not just browser, but also OS, device and even the chips inside are going to be tailored for AI agents running locally.

Re:Show HN: I've built a locally running Perplexity clone by manojlds on Thursday April 04, 2024 @03:31AM (#39927541)
If everyone consumes like that what's even the incentive for content creators?
Re:Show HN: I've built a locally running Perplexity clone by visarga on Thursday April 04, 2024 @07:43AM (#39929070)
If content creators can't find anything that is uniquely human and cannot be made by AI, then maybe they are not creative enough for the job. The thing about generative AI is that it can take context, you can put a lot or very little guidance in it. The more you specify, the more you can mix your own unique sauce in the final result.

I personally use AI for text style changes, as a summarizer of ideas and as rubber duck, something to bounce ideas off of. It's good to get ideas flowing and sometimes can help you realize things you missed, or frame something better than you could.

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:37AM (#39927881)
I didn't run into a lot of timeouts while using it myself, but you would probably need another search source if you plan to host this service for multiple users at the same time.

There are projects like flareresolverr which might be interesting

Re:Show HN: I've built a locally running Perplexity clone by mark_l_watson on Wednesday April 03, 2024 @09:03PM (#39925357)
Your project looks very cool. I had on my ‘list’ to re-learn Typescript (I took a TS course about 5 years ago, but didn’t do anything with it) so I just cloned your repo so I can experiment with it.

EDIT: I just noticed that most of the code is Go. Still going to play with it!

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:35AM (#39927869)
Thanks :). Yea only the web part is typescript and I really wouldn't recommend to learn from my typescript haha
Re:Show HN: I've built a locally running Perplexity clone by ivolimmen on Thursday April 04, 2024 @12:39AM (#39926588)
"normal consumer GPU"... well mine is a 4GB 6600.. so I guess that varies.
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:35AM (#39927864)
Sorry it wasn't my intention to gatekeep, but my 300€ card really is on the low end of LLM Things
Re:Show HN: I've built a locally running Perplexity clone by FezzikTheGiant on Thursday April 04, 2024 @01:28AM (#39926857)
If you're open to it, it would be great if you could make a post explaining how you built this. Even if it's brief. Trying to learn more about this space and this looks pretty cool. And ofc, nice work!
Re:Show HN: I've built a locally running Perplexity clone by Nischalj10 on Thursday April 04, 2024 @04:03AM (#39927714)
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @05:22AM (#39928192)
guys, i didn't thought there would be this much interest in my project haha. I feel kinda bad for just posting it in this state haha. I would love to make a more detailed post on how it works in the future (keep an eye on the repo?)
Re:Show HN: I've built a locally running Perplexity clone by FezzikTheGiant on Thursday April 04, 2024 @03:29PM (#39934764)
look forward to it. this is really cool
Re:Show HN: I've built a locally running Perplexity clone by FezzikTheGiant on Thursday April 04, 2024 @03:30PM (#39934777)
Thanks! As a CS student interested in learning more about this space how do you recommend I get started? I'm pretty early in my education so I kind of want to learn how to drive the car for now and learn how the engine works more formally later, if you know what I mean.
Re:Show HN: I've built a locally running Perplexity clone by keefle on Thursday April 04, 2024 @03:25AM (#39927521)
Wonderful work!

is it possible to make it only use a subset of the web? (Only sites that I trust and think are relevant to producing an accurate answer), and are there ways to possibly make it work offline on pre installed websites? (wikipedia, some other wikis and possibly news sites that are archived locally), and how about other forms of documents? (books and research papers as pdfs)

Re:Show HN: I've built a locally running Perplexity clone by nemoniac on Thursday April 04, 2024 @03:59AM (#39927690)
Llocalsearch uses searxng which has a feature to blacklist/whitelist sites for various purposes.
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @05:18AM (#39928170)
also a great idea to expose this to the frontend. thanks :)
Re:Show HN: I've built a locally running Perplexity clone by kidintech on Thursday April 04, 2024 @03:32AM (#39927548)
Seconded. I tried to do this many years ago for my dissertation and failed, but this would be a dream of mine.
Re:Show HN: I've built a locally running Perplexity clone by robertlagrant on Thursday April 04, 2024 @03:40AM (#39927588)
Would it not be possible to create a search engine that only crawls certain sites?
Re:Show HN: I've built a locally running Perplexity clone by kidintech on Thursday April 04, 2024 @03:43AM (#39927608)
I was most interested in the offline aspect of it, which I wouldn't know where to even start with if I were to fork.

How do you parse and efficiently store large, unstructured information for arbitrary, unstructured queries?

Re:Show HN: I've built a locally running Perplexity clone by stavros on Thursday April 04, 2024 @05:12AM (#39928127)
You put it in a search server, like ElasticSearch or Meili.
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @05:18AM (#39928166)
uhhhh both ideas are great, would you like to turn them into github issues? i will definitely look into both of them :)
Re:Show HN: I've built a locally running Perplexity clone by d-z-m on Thursday April 04, 2024 @05:36AM (#39928282)
any plans to support other backends besides ollama?
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @08:02AM (#39929269)
Sure (if they are openai api compatible i can add them within minutes) otherwise I'm open for pull requests :)

Also, i don't own an Nvidia Card or Windows / MacOS

Re:Show HN: I've built a locally running Perplexity clone by ziziman on Thursday April 04, 2024 @10:02AM (#39930529)
To scrape the websites, do you just blindly cut all of the HTML into defined size chunks or is there some more sophisticated logic to extract text of interest ?

I'm wondering because most news websites now have a lot of polluting elements like popups, would they also go into the database ?

Re:Show HN: I've built a locally running Perplexity clone by totolouis on Thursday April 04, 2024 @10:33AM (#39930923)
If you look at the vector handler in his code, he is using blue Monday sanitizer and doing some "replaceAll".

So I think there may be some useless data in the vector, but that may not be a issue since it is coming from multiple sources (for simple question at least)

Re:Show HN: I've built a locally running Perplexity clone by hanniabu on Thursday April 04, 2024 @11:18AM (#39931501)
This is awesome, would love if there were executable files where these dependencies are needed. That would make it wayyyy more accessible rather than just to those that know how to use the command line and resolve dependencies (yes, even docker runs into that when fighting the local system).
Re:Show HN: I've built a locally running Perplexity clone by fnetisma on Wednesday April 03, 2024 @07:48PM (#39924830)
This is really neat! I have questions:

“Needs tool usage” and “found the answer” blocks in your infra, how are these decisions made?

Looking at the demo, it takes a little time to return results, from the search, vector storage and vector db retrieval, which step takes the most time?

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @07:56PM (#39924890)
Thanks :)

Die LLM makes these decisions on its own. If it writes a message which contains a tool call (Action: Web search Action Input: weight of a llama) the matching function will be executed and the response returned to the LLM. It's basically chatting with the tool.

You can toggle the log viewer on the top right, to get more detail on what it's doing and what is taking time. Timing depends on multiple things: - the size of the top n articles (generating embeddings for them takes some time) - the amount of matching vector DB responses (reading them takes some time)

Re:Show HN: I've built a locally running Perplexity clone by dcreater on Thursday April 04, 2024 @01:53AM (#39927000)
> Die LLM

You mean the? The German is bleeding through haha

Re:Show HN: I've built a locally running Perplexity clone by rzzzt on Thursday April 04, 2024 @02:13AM (#39927090)
Wolfenstein 3D did it first! And then The Simpsons as well.
Re:Show HN: I've built a locally running Perplexity clone by pants2 on Wednesday April 03, 2024 @08:02PM (#39924924)
It says it's a "locally running search engine" - but not sure how it finds the sites and pages to index in the first place?
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @08:11PM (#39925003)
Yea I guess that's misleading, I should probably change that. I was referring to the LLM part as locally running. Indexing is still done by the big guys and queried using searxng
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @08:18PM (#39925050)
Just to clarify, it wasn't my intention to be misleading
Re:Show HN: I've built a locally running Perplexity clone by lavela on Thursday April 04, 2024 @04:55AM (#39928019)
What would be your current recommendation on how to create a vector db from local files that would work with LLocalSearch?
Re:Show HN: I've built a locally running Perplexity clone by keyle on Wednesday April 03, 2024 @06:50PM (#39924346)
Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on).

I too use local 7b open-hermes and it's really good.

Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Wednesday April 03, 2024 @08:10PM (#39924989)
I'm just starting to get into downloading and testing models using llama.cpp and I'm curious which model you're actually using, since they seem to come in varying levels of quantization. Is this [0] the model page for the one you're using, or should I be looking somewhere else? What is the actual file name of the model you're using?

[0] https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GG...

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @06:54PM (#39924375)
Thanks :). It's just a lot of prompting and string parsing. There are models like "Hermes-2-Pro-Mistral" (the one from the video) which are trained to work with function signatures and outputting structured text. But at the end it's just strings in > strings out, haha. But its fun (and sometimes frustrating) to use LLMs for flow control (conditions, loops...) inside your programs.
Re:Show HN: I've built a locally running Perplexity clone by davidcollantes on Wednesday April 03, 2024 @08:08PM (#39924970)
Got a link for that one? I have found a few with Hermes-2-Mistral in the name.
Re:Show HN: I've built a locally running Perplexity clone by keyle on Wednesday April 03, 2024 @07:22PM (#39924647)
Wow, I didn't know about "Hermes 2 Pro - Mistral 7B", cheers!
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @07:35PM (#39924735)
It's my go to "structured text model" atm. Try "starling-ml-beta" (7b) for some very impressive chat capabilities. I honestly think that it outperforms GPT3 half the time.
Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Wednesday April 03, 2024 @08:12PM (#39925006)
Sorry to repeat the same question I just asked the other commenter in this thread, but could you link the model page and recommend a specific level of quantization for the models you've referenced? I'd love to play with these models and see what you're talking about.
Re:Show HN: I've built a locally running Perplexity clone by BOOSTERHIDROGEN on Wednesday April 03, 2024 @08:19PM (#39925059)
Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Wednesday April 03, 2024 @08:31PM (#39925125)
Thank you — from that page, at the bottom, I was able to find this link to what I think are the quantized versions

https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-...

If you have the time, could you explain what you mean by "Q5 is minimum"? Did you determine that by trying the different models and finding this one is best, or did someone else do that evaluation, or is that just generally accepted knowledge? Sorry, I find this whole ecosystem quite confusing still, but I'm very new and that's not your problem.

Re:Show HN: I've built a locally running Perplexity clone by BOOSTERHIDROGEN on Thursday April 04, 2024 @01:59AM (#39927029)
It's the best balance if you have limited compute performance.
Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Thursday April 04, 2024 @12:51PM (#39932897)
Thank you
Re:Show HN: I've built a locally running Perplexity clone by d-z-m on Thursday April 04, 2024 @05:13AM (#39928129)
Talking GGUF, Usually the higher you can afford to go wrt. quantization(e.g. Q5 is better than Q4, etc), the better. A Q6_K has minimal performance loss from the Q8, so in most cases if you can fit a Q6_K it's recommended to just use that. TheBloke's READMEs[0] usually have a good table summarizing each quantization level.

If you're RAM constrained, you'll also have to make trade-offs about the context length. e.g. you could have 8 GB RAM and a Q5 quant with shorter context, vs Q3 with longer, etc.

[0]:https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF

Re:Show HN: I've built a locally running Perplexity clone by peter_l_downs on Thursday April 04, 2024 @12:51PM (#39932899)
Thank you!
Re:Show HN: I've built a locally running Perplexity clone by madacol on Friday April 05, 2024 @09:31AM (#39942149)
Have you considered using grammar sampling?
Re:Show HN: I've built a locally running Perplexity clone by viksit on Wednesday April 03, 2024 @07:08PM (#39924495)
curious what hardware you use? and is any of this runnable on an m1 laptop?
Re:Show HN: I've built a locally running Perplexity clone by keyle on Wednesday April 03, 2024 @07:11PM (#39924525)
Absolutely, 7B will run comfortably on 16GB of RAM and most consumer level hardware. Some of the 40B run on 32GB, but it depends on the model I found (GGUF, crossing fingers help).

I ran this originally on a M1 with 32GB, I run this on an Air M2 with 16GB (and mac mini M2 32GB), no problem.

I use llama.cpp with a SwiftUI interface (my own), all native, no scripts python/js/web.

7b is obviously less capable but the instant response makes it worth exploring. It's very useful as a Google search replacement that is instantly more valuable, for general questions, than dealing with the hellscape of blog spam ruling Google atm.

Note, for my complex code queries at $dayjob where time is of the essence, I still use GPT4 plus, which is still unmatched imho, without running special hardware at least.

Re:Show HN: I've built a locally running Perplexity clone by regularfry on Thursday April 04, 2024 @06:06AM (#39928449)
I've been occasionally using a 7b Q4 quant on llama.cpp on an 8GB M1. It's usable, if not amazing.
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Wednesday April 03, 2024 @07:33PM (#39924720)
Depends on your m1 specs, but should definitely be able to run a 7b model (at least with some quantization).
Re:Show HN: I've built a locally running Perplexity clone by windexh8er on Wednesday April 03, 2024 @08:59PM (#39925306)
Have you looked into tools like CrewAI [0]?

[0] https://www.crewai.io/

Re:Show HN: I've built a locally running Perplexity clone by pcthrowaway on Wednesday April 03, 2024 @08:30PM (#39925121)
Now if someone can hook this into Plandex (also shared today - https://news.ycombinator.com/item?id=39918500) to make a tool that enables you to collaborate with AI without any of your code leaving your computer, that would be amazing!
Re:Show HN: I've built a locally running Perplexity clone by ldjkfkdsjnv on Wednesday April 03, 2024 @08:39PM (#39925177)
The big secret about perplexity is they havent done much beyond using off the shelf models
Re:Show HN: I've built a locally running Perplexity clone by ggnore7452 on Thursday April 04, 2024 @12:33AM (#39926550)
I've been working on a small personal project similar to this and agree that replicating the overall experience provided by Perplexity.ai, or even improving it for personal use, isn't that challenging. (The concerns of scale or cost are less significant in personal projects. Perplexity doesn't do too much planning or query expansion, nor does it dig super deep into the sources afaik)

I must say, though, that they are doing a commendable job integrating sources like YouTube and Reddit. These platforms benefit from special preprocessing and indeed add value.

Re:Show HN: I've built a locally running Perplexity clone by KuriousCat on Wednesday April 03, 2024 @10:13PM (#39925807)
How did they secure funds in that case?
Re:Show HN: I've built a locally running Perplexity clone by basbuller on Thursday April 04, 2024 @12:14AM (#39926459)
That is probably exactly why they got funding. You can sell it as focus on adding new features and leveraging the best available tools before reinventing the wheel.

They do train their own models now, but for about a year they just forwarded calls to models like gpt3.5T. You still have the option to use models not trained by perplexity.

Re:Show HN: I've built a locally running Perplexity clone by hackernewds on Thursday April 04, 2024 @12:29AM (#39926531)
which is why their engagement and model responses suck. the other competitors are far better

C.ai and Pi comes to mind

Re:Show HN: I've built a locally running Perplexity clone by basbuller on Thursday April 04, 2024 @12:41AM (#39926597)
Wait, are you directly comparing Perplexity and C.ai or Pi? Perplexity is a search engine, Pi is a chatbot, and C.ai is roleplay? Their value propositions are very different
Re:Show HN: I've built a locally running Perplexity clone by KuriousCat on Thursday April 04, 2024 @12:55AM (#39926683)
I still don't get it. What was the USP here? What is the allure in it for the investors?
Re:Show HN: I've built a locally running Perplexity clone by code51 on Thursday April 04, 2024 @04:10AM (#39927731)
Simple, looking at the mirror and saying "Google-killer" firmly 3 times everyday.
Re:Show HN: I've built a locally running Perplexity clone by msp26 on Thursday April 04, 2024 @05:39AM (#39928297)
Pretraining and even finetuning (to a good extent) is overrated and you can create plenty of value without it.
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @08:05AM (#39929305)
I assume the same, feels like their product is just summarizing the top n results? I wouldn't need the whole vector db thing, if local models (or hardware) would be able to run with a context of this size.
Re:Show HN: I've built a locally running Perplexity clone by xydac on Wednesday April 03, 2024 @08:51PM (#39925249)
This is cool, haven't run this yet but seems really promising. Am thinking how this can be a super useful to hook with internal corporate search engines and then get answers from that.

Good to see more of these non API key products being built (connected to local llms)

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @06:42AM (#39928653)
I might try to hook this into our internal confluence, shouldn't be a problem
Re:Show HN: I've built a locally running Perplexity clone by darby_eight on Wednesday April 03, 2024 @09:29PM (#39925561)
Perplexity seems to be a chatbot competitor.
Re:Show HN: I've built a locally running Perplexity clone by hackernewds on Thursday April 04, 2024 @12:32AM (#39926546)
what does this have to do with Perplexity? it should reference the underlying models used instead
Re:Show HN: I've built a locally running Perplexity clone by vishnumohandas on Thursday April 04, 2024 @12:37AM (#39926583)
The UX is comparable.
Re:Show HN: I've built a locally running Perplexity clone by smcleod on Thursday April 04, 2024 @03:05PM (#39934525)
UI does show and let you select the underlying model
Re:Show HN: I've built a locally running Perplexity clone by firtoz on Thursday April 04, 2024 @01:40AM (#39926928)
Excellent work! I plan to use it with existing LLMs tbh, but great to see it working locally also! Thank you so much for sharing. I love the architecture.
Re:Show HN: I've built a locally running Perplexity clone by hubraumhugo on Thursday April 04, 2024 @03:10AM (#39927417)
Excellent work! Cool side projects like that will eventually help you get hired by a top startup or may even lead to building your own.

I can only encourage other makers to post their projects on HN and put them out into the world.

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:42AM (#39927926)
Yea it's also quite fulfilling to see people likening something you've put some work into :)
Re:Show HN: I've built a locally running Perplexity clone by ml-anon on Thursday April 04, 2024 @03:14AM (#39927432)
Did you really make a perplexity clone if you didn’t spend more time promoting yourself on Twitter and LinkedIn than on the engineering?
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:43AM (#39927931)
Ah damn I forgot about getting some VC money
Re:Show HN: I've built a locally running Perplexity clone by ProllyInfamous on Thursday April 04, 2024 @12:59PM (#39933000)
This demo will land you more important things than "just" VC money.

Extremely impressive, cannot wait to actually implement this on my M2Pro (mac).

Re:Show HN: I've built a locally running Perplexity clone by arflikedog on Thursday April 04, 2024 @03:19AM (#39927471)
A while back you commented on my personal project Airdraw which I really appreciated. This looks awesome and you're well on your way to another banger project - looking forward to toying around with this :)
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:37AM (#39927884)
Uhh yes I was really impressed by your project :)
Re:Show HN: I've built a locally running Perplexity clone by gardenhedge on Thursday April 04, 2024 @06:01AM (#39928412)
Did you just happened to see this post today and notice the username?
Re:Show HN: I've built a locally running Perplexity clone by arflikedog on Thursday April 04, 2024 @11:21AM (#39931553)
unironically yes, I used comments to hot fix a bunch of stuff when I first launched. It's a small world and I thought this was a cool moment
Re:Show HN: I've built a locally running Perplexity clone by noisy_boy on Thursday April 04, 2024 @04:13AM (#39927744)
Would be good if the readme mentions minimum hardware specs to get a reasonably decent performance. E.g. I have a ThinkPad X1 extreme i7 with MaxQ graphics, any hopes of running this on it without completely ruining the performance?
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:41AM (#39927915)
You could run the LLM using your CPU and normal (non video) ram. But that's a lot slower. There are people working on making it a lot faster tho. The bottleneck is the transfer speed between the ram Sticks and the CPU.

Just taking a guess, but I wouldn't expect more than a couple tokens (more or less like syllables) per second. Which is probably to slow, since it has to read a couple thousand per search result.

It's hard to provide minimum requirements, since there are so many edge cases.

Re:Show HN: I've built a locally running Perplexity clone by oysterpingu on Thursday April 04, 2024 @04:23AM (#39927803)
Awesome project! As I newbie myself in everything LLM, where should I start looking to create a similar project than yours? Which resources/projects are good to know about? Thank you for sharing!
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:33AM (#39927853)
I think the easiest entry point would be the python langchain project? It has a lot more documentation and working examples than the golang one I've used :)

If you could tell me more about your goals, I can probably provide a more narrow answer :)

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:31AM (#39927847)
Uhh sorry guys, I was asleep and now the project has like 1k stars haha

I will try my best to catch up with everyone <3

Re:Show HN: I've built a locally running Perplexity clone by nikolayasdf123 on Thursday April 04, 2024 @04:39AM (#39927899)
cool to see Go here
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @06:43AM (#39928655)
langchain go (missed opportunity to call it golang-chain) is nice, but has literally no docs haha
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @08:08AM (#39929347)
btw ollama (the webserver around llamacpp) is also written in golang :)
Re:Show HN: I've built a locally running Perplexity clone by adr1an on Thursday April 04, 2024 @04:50AM (#39927993)
This is so cool! And the fact that you can use Ollama as 'llm backend' makes it sustainable. didn't see how to switch models in the demo, that might be worth to highlight in readme..
Re:Show HN: I've built a locally running Perplexity clone by adr1an on Thursday April 04, 2024 @05:32AM (#39928254)
I have a 'feature request', can we manage which sites are being used by some categories in the frontend? For example, if I build a list of websites and out them under "coding", then I'd like to use those to answer my programming questions. Meanwhile, I'd like to add an "art" category for museum's homepages so that I can ask which year was XYZ painting from. And so on. The current implementation looks like the inter-operability with searing is more static... IDK if searxng has an API to switch those filters or if they can be managed already through 'profiles'.. that kind of thing..
Re:Show HN: I've built a locally running Perplexity clone by gorbypark on Thursday April 04, 2024 @05:07AM (#39928096)
I if a quick poke through the source and it seems like there’s not much reason this couldn’t run on macOS? It seems that ollama is doing the inference and then there’s a go binary doing everything else? I might give it a go and see what happens!
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @06:42AM (#39928647)
sure there are people in the issues how got it working on macos. docker networking was the only problem :)
Re:Show HN: I've built a locally running Perplexity clone by mritchie712 on Thursday April 04, 2024 @07:21AM (#39928901)
I have it running on my Mac right now, took < 2 minutes (had to manually download one of the ollama models)
Re:Show HN: I've built a locally running Perplexity clone by siborg on Thursday April 04, 2024 @05:21AM (#39928180)
Exciting project. Trying to install it but running into some issues with searxng. Anyone else?
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @05:25AM (#39928206)
please tell me about the problem or open an issue :)
Re:Show HN: I've built a locally running Perplexity clone by wg0 on Thursday April 04, 2024 @05:38AM (#39928291)
In five year's time - by 2030, I foresee that lots of inference would be happening on local machines with models being downloaded on demand. Think docker registry of AI models which is pretty much Hugging Face already there.

This all would be due to optimisations within model inference code and techniques, hardware and packaging of software like the above.

Don't see billion dollar valuations for lots of AI startups out there to materialise into anything.

Re:Show HN: I've built a locally running Perplexity clone by openquery on Thursday April 04, 2024 @05:46AM (#39928327)
> I foresee that lots of inference would be happening on local machines with models being downloaded on demand

Why? It's much more efficient to have centralized special purpose hardware to run enormous models and then ship the comparatively small result over the internet.

By analogy, you don't have a search engine running on your phone right?

Re:Show HN: I've built a locally running Perplexity clone by Sammi on Thursday April 04, 2024 @05:59AM (#39928400)
You currently can't have a search engine running locally on your phone. Google search is possible the single largest c++ program every built. And nevermind the storage needs...

But in a few years we might be able to have LLMs running on our phones that work just as well if not better. Of couse as you mention the LLMs running on large servers might still be much more powerfull, but the local ones might be powerfull enough.

Re:Show HN: I've built a locally running Perplexity clone by vachina on Thursday April 04, 2024 @05:51AM (#39928358)
A more appropriate analogy would be driving your own car vs. taking the bus.
Re:Show HN: I've built a locally running Perplexity clone by bufferoverflow on Thursday April 04, 2024 @11:24PM (#39938401)
No, a more appropriate analogy would be driving your own billion-dollar super-yacht vs driving your own car.

Will not happen any time soon. Consumer hardware can't even run GPT-4 locally, and won't be able for a looong time. Each GPT-4 instance runs on 8 A100. The cost of such system is ~$81K. Not even in the ballpark of what most consumers can afford.

Re:Show HN: I've built a locally running Perplexity clone by dns_snek on Thursday April 04, 2024 @07:49AM (#39929134)
> Why?

Privacy, security, latency, offline availability, access to local data and services running on the device, just to name a few.

Re:Show HN: I've built a locally running Perplexity clone by ilc on Thursday April 04, 2024 @10:58AM (#39931268)
Big Tech + Countries: Those all sound like great reasons to centralize all access to AIs!
Re:Show HN: I've built a locally running Perplexity clone by ThinkBeat on Thursday April 04, 2024 @06:23AM (#39928533)
"640kb will be enough for everyone." (Gates)

I think that the models will evolve and grow as more powerful compute/hardware comes out.

You may be able to run scaled down n versions of what state of the art now, but by then the giant models will have grown in size and in required compute.

The 6 year old models will be retro computingish.

Somewhat like how you can play 6 year old games on a new powerful PC but by then the new huge games will no longer play well on your Old mach

Re:Show HN: I've built a locally running Perplexity clone by lobocinza on Thursday April 04, 2024 @11:00AM (#39931283)
There will be demand and supply for both cases.
Re:Show HN: I've built a locally running Perplexity clone by bufferoverflow on Thursday April 04, 2024 @11:19PM (#39938375)
Unfortunately training is insanely expensive. StabilityAI is struggling with staying alive. And Anthropic wants to spend $100 BILLION building a supercomputer just for training.

That's why I think these private companies will have the best AIs for many decades.

Re:Show HN: I've built a locally running Perplexity clone by BrutalCoding on Thursday April 04, 2024 @05:46AM (#39928321)
That’s a great project you pulled off. From the time I starred it (10-12h ago I think), and upon re-checking this post, you gained 500+ stars lol.

Visualized in a chart with star-history: https://star-history.com/#nilsherzig/LLocalSearch

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @06:01AM (#39928416)
haha thanks for the chart link. i woke up with 1k more than it had yesterday, im kinda stressed out
Re:Show HN: I've built a locally running Perplexity clone by sroussey on Thursday April 04, 2024 @11:56AM (#39932048)
Ah, that’s a nice chart generator. Will have to use if I ever get any, lol.
Re:Show HN: I've built a locally running Perplexity clone by gardenhedge on Thursday April 04, 2024 @05:58AM (#39928397)
Completely locally running search engine.. that queries the Internet
Re:Show HN: I've built a locally running Perplexity clone by sebzim4500 on Thursday April 04, 2024 @06:31AM (#39928573)
Whenever I see these projects I always find reading the prompts fascinating.

> Useful for searching through added files and websites. Search for keywords in the text not whole questions, avoid relative words like "yesterday" think about what could be in the text. > The input to this tool will be run against a vector db. The top results will be returned as json.

Presumably each clarification is an attempt to fix a bug experienced by the developer, except the fix is in English not in Go.

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @06:39AM (#39928623)
haha yea pretty much, its amazing (and frustrating) how much of the programs "performance" depends on these prompts
Re:Show HN: I've built a locally running Perplexity clone by htrp on Thursday April 04, 2024 @08:17AM (#39929440)
Our current state of the art

also love your last commit

>fix: copilot is stupid and i should not blindly trust it

>https://github.com/nilsherzig/LLocalSearch/commit/9f45e24f15...

Everything wrong with code gen in a nutshell

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @09:32AM (#39930197)
Yea im kinda stressed out to get it working for everyone haha. I would have caught that under different conditions. I'm a big e2e tests guy haha
Re:Show HN: I've built a locally running Perplexity clone by andrewfromx on Thursday April 04, 2024 @06:37AM (#39928617)
searXNGDomain := os.Getenv("SEARXNG_DOMAIN")

I see this but what search engine lets you get results in json for free?

Re:Show HN: I've built a locally running Perplexity clone by andrewfromx on Thursday April 04, 2024 @06:39AM (#39928621)
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @06:40AM (#39928635)
Those arent search results tho, that's just duckduckgo internal things like "similar queries"
Re:Show HN: I've built a locally running Perplexity clone by andrewfromx on Thursday April 04, 2024 @06:44AM (#39928663)
oh they must be hitting their own internal api with format=json but what is the datasource?
Re:Show HN: I've built a locally running Perplexity clone by andrewfromx on Thursday April 04, 2024 @06:46AM (#39928673)
Re:Show HN: I've built a locally running Perplexity clone by sgt on Thursday April 04, 2024 @07:18AM (#39928873)
Speaking of LLM's... here's my "dear lazyweb" to HN:

What would be the best self hosted option to build sort of a textual AI assistant into your app? Preferably something that I can train myself over time with domain knowledge.

Re:Show HN: I've built a locally running Perplexity clone by traverseda on Thursday April 04, 2024 @07:33AM (#39928997)
Fine tuning on your own knowledge probably isn't what you want to do, you probably want to do retrieval aided generation instead. Basically a search engine on some local documents, and you put the results of the search into your prompt. The search engine uses the same vector space as your language model as its index, so the results should be highly relevant to whatever the prompt is.

I'd start with "librechat" and mistral, so far that's one of the best chat interfaces and has good support for self hosting. For the actual model runner, ollama seems to be the way to go.

I believe it's built on "langchain", so you can switch to that when it makes sense to. When you've tested all your queries and setup with librechat, know that librechat is a wrapper around "langchain".

I'd start by testing the workflow in librechat, and if librechat's API doesn't do what you want, well I've always found fastAPI pleasant to work with.

---

Less for your use case, and more in-general. I've been assessing a lot of LLM interfaces lately, and the weird porn community has some really powerful and flexible interfaces. With sillytavern you can set up multiple agents, have one agent program, another agent critique, and a third asses it for security concerns. This kind of feedback can help catch a lot of LLM mistakes. You can also go back and edit the LLM's response, which can really help. If you go back and edit an LLM message to fix code or change variable names, it will tend to stick with those decisions. But those interfaces are still very much optimized for "Role playing".

Recommend keeping an eye on https://www.reddit.com/r/LocalLLaMA/

Re:Show HN: I've built a locally running Perplexity clone by sgt on Friday April 05, 2024 @07:23AM (#39941033)
Thanks - will check out librechat etc. It's interesting that fine tuning is no longer the thing to do. I am not clear on how one connects librechat to local data but am sure I will when I dive deeper into this.
Re:Show HN: I've built a locally running Perplexity clone by madeofpalk on Thursday April 04, 2024 @07:28AM (#39928955)
> Q: is chrome on ios powered by safari

> According to the sources provided, Chrome on iOS is not powered by Safari. Google's Chrome uses the Blink engine, while Safari uses the WebKit engine.

I find it amusing how when people show off their LLM projects their examples are always of it failing, and providing a bad answer.

Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @08:03AM (#39929290)
Well i don't indent to get money from people, so i guess showing real results isnt a "problem".

Besides i think the following sentences arent wrong? Its just a 7b model give it some slack haha

Re:Show HN: I've built a locally running Perplexity clone by madeofpalk on Thursday April 04, 2024 @09:05AM (#39929900)
No, sure. I just think it's funny how it constantly happens, from opensource, free, or commercial projects. No one seems to be immune to 'telling on themselves'.
Re:Show HN: I've built a locally running Perplexity clone by aagha on Thursday April 04, 2024 @02:15PM (#39934010)
According to Crunchbase [0], Perplexity has raised over $100M.

You built this in your spare time?

The following things jump out to me:

- How much a hype cycle invites insane amounts of money - How trash the entire VC world is during a hype cycle - What an amazing thing ingenuity and passion are

Great job!

0 - https://www.crunchbase.com/organization/perplexity-ai

Re:Show HN: I've built a locally running Perplexity clone by high_priest on Thursday April 04, 2024 @05:18PM (#39935913)
Why call it "Perplexity clone" when this is much more than what Perplexity offers?

Btw. This is the first time I hear about Perplexity, which after 10 minutes of experimentation, looks like a worse clone of Phind.

Re:Show HN: I've built a locally running Perplexity clone by frantic2821 on Friday April 05, 2024 @09:19AM (#39942010)
don't stop working on this!!
Re:Show HN: I've built a locally running Perplexity clone by XCSme on Friday April 05, 2024 @03:45PM (#39946351)
Can you integrate this type of search in Ollama?
Re:Show HN: I've built a locally running Perplexity clone by monkeydust on Wednesday April 17, 2024 @09:20AM (#40064200)
This looks great, can I get a series of agents with clearly defined roles working together on a problem akin to Autogen?