News for hackers. Stuff that matters.

A comment on I just finished crawling 5.19B web pages, Ask Me Anything.

Re:I just finished crawling 5.19B web pages, Ask Me Anything by joshpen188 on Wednesday April 19, 2017 @07:12PM (#14152613)
Why didn't you use common crawl instead?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack_2 on Wednesday April 19, 2017 @07:40PM (#14152761)
For our purposes Common Crawl's corpus was missing too many websites (possibly due to robots.txt configs of websites) Also we needed some deep coverage which CC could not provide.
[ Parent | on HN ]