News for hackers. Stuff that matters.

A comment on Triv.io donates URL index to Common Crawl. Parent comment

Re:Triv.io donates URL index to Common Crawl by srobertson on Tuesday January 08, 2013 @06:45PM (#5029243)
That would be great. Out of curiosity, mind describing what you'd use such a list for?
[ Parent | on HN ]
Re:Triv.io donates URL index to Common Crawl by rb2k_ on Wednesday January 09, 2013 @10:25PM (#5035188)
Usually as a starting point for a crawler when checking statistics about the web.

Crawling is much faster if you don't have to "spider" all the links and check if you've already visited them or not.

With a big enough list, you can just iterate over those domains. (average number of links on a website, how often does javascript framework x vs y get used, how many sites have an HTML5 doctype yet, ...)

[ Parent | on HN ]