News for hackers. Stuff that matters.
Re:Ask HN: Has anyone ever crawled over a billion pages? How... by cdnsteve on Wednesday September 14, 2016 @05:03PM (#12501192)
I think it would be valuable to have an open dataset of a raw crawl index. It could be distributed via academic torrents or partner with a hosting provider.

The real innovation won't be in crawling but in working on the index, filtering it, organizing it, trying sort algorithms and learning.

If this was available and gained popularity I could see competition in search again.

There are no comments on this comment yet.