News for hackers. Stuff that matters.

A comment on Show HN: A search engine that doesn't track you, where users vote for results. Parent comment

Re:Show HN: A search engine that doesn't track you, where us... by brentadamson on Wednesday May 02, 2018 @12:59PM (#16978916)
We need more links indexed. The crawler that I wrote is pretty slow and right now we don't have a lot of documents indexed. I will be replacing the crawler I wrote with a faster one soon. Until then I have been using !bangs.
[ Parent | on HN ]
Re:Show HN: A search engine that doesn't track you, where us... by ericst on Thursday May 03, 2018 @02:00AM (#16983662)
Have you ever heard of https://commoncrawl.org/? Maybe it helps you to bootstrap.

It's cool to see some new things happening in the search engine world.

[ Parent | on HN ]
Re:Show HN: A search engine that doesn't track you, where us... by brentadamson on Thursday May 03, 2018 @11:02AM (#16986645)
I've considered that but didn't really know anything about the organization and how long they'll be around. We're gonna give Colly a try: https://github.com/gocolly/colly.
[ Parent | on HN ]