News for hackers. Stuff that matters.

A comment on I just finished crawling 5.19B web pages, Ask Me Anything. Parent comment

Re:I just finished crawling 5.19B web pages, Ask Me Anything by savethefuture on Wednesday April 19, 2017 @07:02PM (#14152559)
How long did it take? What type of data did you record?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack on Wednesday April 19, 2017 @07:08PM (#14152586)
It took us about 13 days. We recorded reources of all types: text/, image/, application/*

As one would expect the vast majority of data recorded is text/* (html,...)

[ Parent | on HN ]