News for hackers. Stuff that matters.

A comment on Common Search – nonprofit search engine for the Web. Parent comment

Re:Common Search – nonprofit search engine for the Web by struct on Monday March 14, 2016 @12:57PM (#11283907)
Some of my primary interests are building websites that can analyse other websites and for that, I need a keyword index and access to the underlying crawl data (as in, a response that can point me to the exact file offsets that contain the pages). Think [1] but with keywords instead of URLs.

[1] http://index.commoncrawl.org/

[ Parent | on HN ]
Re:Common Search – nonprofit search engine for the Web by sylvinus on Monday March 14, 2016 @01:27PM (#11284097)
Thanks!

We probably won't expose the underlying crawl data ourselves but being able to reference it just like [1] does is indeed important.

[ Parent | on HN ]