News for hackers. Stuff that matters.
Re:Custom Search Engine Built on Searx by rolisz on Monday September 13, 2021 @08:20AM (#28510538)
I'm interested in something similar. What I've tried is to set up https://yacy.net/ to index only those websites I care about. I set it up on my home NAS and I configured it to crawl Python, sklearn, and some blogs and it took around 1 hour.

Searching through the results was quite fast, but I found the results a bit lacking. Maybe with more tuning I could have obtained better answers, but while searching for things in the Python standard library, I would get lots of noise from other places where that function is used.

Re:Custom Search Engine Built on Searx by wcerfgba on Monday September 13, 2021 @08:46AM (#28510795)
Yes I was originally considering crawling and indexing on my local machine as well but once I realised I could build this with Searx and leave all of that work to Bing/Google/DDG/... it removed a lot of complexity from the project, and also means I don't have to worry about optimising the quality of the results. That said, there is scope to do more work around plugging all the result sets together: Searx's approach is quite naive and doesn't work as well in this use case as it does in it's original/intended use case.
Re:Custom Search Engine Built on Searx by 6510 on Monday September 13, 2021 @10:19AM (#28511826)
YaCy does full text search and indexes all pages.

edit: But it works best for just a single set of websites. I wouldn't know how to build multiple indexes.