Re:We can do better than DuckDuckGo
by edgyquant on Tuesday November 17, 2020 @10:38PM
(#25132935)
Aside from this a big reason to build this is it seems a lot simpler than writing a giant web crawler ala google and thus is a good target for an open source solution. Which is the biggest problem with duck duck go.
[ Parent
| on HN ]
Do you have thoughts on implementing the distributed search?
I'm thinking about playing around with this in my spare time, but that part seems the hardest to do.
Don't start from scratch, take a look at yacy (https://yacy.net/) that already does most of what is discussed here
I would think everyone would run their own “crawler” but maybe you could use a ledger and delegate sites to different workers. If you’re whitelisting sites you could maybe only crawl a link or two deep.
It’d take a lot of cycles while small but if you get a network growing you could even have sub-networks with their own whitelist additions (and every user has a blacklist.)