I just finished crawling 5.19B web pages, Ask Me Anything
I WAS JUST RATE LIMITED BY HN, SO IM GOING TO ANSWER YOUR QUESTIONS UNDER A NEW ACCOUNT: dor_jack_2
dor_jack
Re:Ask HN: What are best tools for web scraping?
on Wednesday November 15, 2017 @01:51PM (#15706600)
If you need to perform a web-scale crawl I strongly recommend https://www.mixnode.com.
Re:I just finished crawling 5.19B web pages, Ask Me Anything
on Wednesday April 19, 2017 @08:54PM (#14153126)
The platform we used provided their own seedlist and took it from there.
Re:I just finished crawling 5.19B web pages, Ask Me Anything
on Wednesday April 19, 2017 @08:49PM (#14153104)
It was an all inclusive deal:
420 TB at 0.06 per GB = $25,804
Re:I just finished crawling 5.19B web pages, Ask Me Anything
on Wednesday April 19, 2017 @07:11PM (#14152601)
If our company approves I would like to publish some general statistics that may be of interest to others.
Re:I just finished crawling 5.19B web pages, Ask Me Anything
on Wednesday April 19, 2017 @07:08PM (#14152586)
It took us about 13 days.
We recorded reources of all types: text/, image/, application/*
As one would expect the vast majority of data recorded is text/* (html,...)
Re:I just finished crawling 5.19B web pages, Ask Me Anything
on Wednesday April 19, 2017 @07:07PM (#14152580)
We are processing the data as we speak. However the movement of technology based on where your company is based is truly incredible.
Will update this in a few days with more data.
Re:I just finished crawling 5.19B web pages, Ask Me Anything
on Wednesday April 19, 2017 @07:06PM (#14152573)
Our company is in the Marketing Intelligence (MI) industry. We needed to measure the penetration of multiple technologies in different countries.