News for hackers. Stuff that matters.

A comment on I just finished crawling 5.19B web pages, Ask Me Anything.

Re:I just finished crawling 5.19B web pages, Ask Me Anything by dm_i386 on Wednesday April 19, 2017 @07:22PM (#14152674)
What tools did you use? What had to be custom-written and why?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack_2 on Wednesday April 19, 2017 @07:38PM (#14152751)
We tried a bunch of technologies like Nutch, Heritrix, Storm Crawler, ... eventually settled on Mixnode and since it's a 'cloud platform' we didn't really have to change anything.

As for processing the data we crawled, we are using ArchiveSpark (https://github.com/helgeho/ArchiveSpark)

Also, Mixnode defaults on Amazon S3 for storage which was ok with us since we're using EC2 for processing the results.

[ Parent | on HN ]