dor_jack_2
Re:I just finished crawling 5.19B web pages, Ask Me Anything
on Wednesday April 19, 2017 @07:43PM (#14152778)
Loop/spam prevention was done by mixnode, I'm not sure how they do it.
The data does not follow a DFS or BFS pattern so pages/site varies greatly by a host's server capacity and anti-crawling configs.
There was a minimum of 10 seconds between followup requests to the same website unless robots.txt had a lower delay. Pretty polite...
Re:I just finished crawling 5.19B web pages, Ask Me Anything
on Wednesday April 19, 2017 @07:40PM (#14152761)
For our purposes Common Crawl's corpus was missing too many websites (possibly due to robots.txt configs of websites) Also we needed some deep coverage which CC could not provide.
Re:I just finished crawling 5.19B web pages, Ask Me Anything
on Wednesday April 19, 2017 @07:38PM (#14152751)
We tried a bunch of technologies like Nutch, Heritrix, Storm Crawler, ... eventually settled on Mixnode and since it's a 'cloud platform' we didn't really have to change anything.
As for processing the data we crawled, we are using ArchiveSpark (https://github.com/helgeho/ArchiveSpark)
Also, Mixnode defaults on Amazon S3 for storage which was ok with us since we're using EC2 for processing the results.