News for hackers. Stuff that matters.
Re:I just finished crawling 5.19B web pages, Ask Me Anything by grzm on Wednesday April 19, 2017 @07:44PM (#14152790)
If you're rate-limited, you can contact the mods via the Contact link in the footer.
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dm_i386 on Wednesday April 19, 2017 @07:22PM (#14152674)
What tools did you use? What had to be custom-written and why?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack_2 on Wednesday April 19, 2017 @07:38PM (#14152751)
We tried a bunch of technologies like Nutch, Heritrix, Storm Crawler, ... eventually settled on Mixnode and since it's a 'cloud platform' we didn't really have to change anything.

As for processing the data we crawled, we are using ArchiveSpark (https://github.com/helgeho/ArchiveSpark)

Also, Mixnode defaults on Amazon S3 for storage which was ok with us since we're using EC2 for processing the results.

[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by maurtinshkreli on Wednesday April 19, 2017 @07:21PM (#14152669)
How much did it cost?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack on Wednesday April 19, 2017 @08:49PM (#14153104)
It was an all inclusive deal: 420 TB at 0.06 per GB = $25,804
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by tlack on Wednesday April 19, 2017 @07:21PM (#14152671)
what did you do to avoid winding up in endless GET url loops? How deep did you get per site, and how did you schedule followup requests?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack_2 on Wednesday April 19, 2017 @07:43PM (#14152778)
Loop/spam prevention was done by mixnode, I'm not sure how they do it.

The data does not follow a DFS or BFS pattern so pages/site varies greatly by a host's server capacity and anti-crawling configs.

There was a minimum of 10 seconds between followup requests to the same website unless robots.txt had a lower delay. Pretty polite...

[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by joshpen188 on Wednesday April 19, 2017 @07:12PM (#14152613)
Why didn't you use common crawl instead?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack_2 on Wednesday April 19, 2017 @07:40PM (#14152761)
For our purposes Common Crawl's corpus was missing too many websites (possibly due to robots.txt configs of websites) Also we needed some deep coverage which CC could not provide.
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by savethefuture on Wednesday April 19, 2017 @07:01PM (#14152551)
What did you discover.
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack on Wednesday April 19, 2017 @07:07PM (#14152580)
We are processing the data as we speak. However the movement of technology based on where your company is based is truly incredible.

Will update this in a few days with more data.

[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by savethefuture on Wednesday April 19, 2017 @07:09PM (#14152589)
That will be an interesting correlation to see different frameworks or tech or even design elements based on geographical location.
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack on Wednesday April 19, 2017 @07:11PM (#14152601)
If our company approves I would like to publish some general statistics that may be of interest to others.
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by savethefuture on Wednesday April 19, 2017 @07:13PM (#14152619)
How did you crawl so many sites, how did you discover them, search engine, ip ranges or another method?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack on Wednesday April 19, 2017 @08:54PM (#14153126)
The platform we used provided their own seedlist and took it from there.
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by savethefuture on Wednesday April 19, 2017 @07:02PM (#14152559)
How long did it take? What type of data did you record?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack on Wednesday April 19, 2017 @07:08PM (#14152586)
It took us about 13 days. We recorded reources of all types: text/, image/, application/*

As one would expect the vast majority of data recorded is text/* (html,...)

[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by itburnslikeice on Wednesday April 19, 2017 @07:01PM (#14152547)
but why?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack on Wednesday April 19, 2017 @07:06PM (#14152573)
Our company is in the Marketing Intelligence (MI) industry. We needed to measure the penetration of multiple technologies in different countries.
[ Parent | on HN ]