News for hackers. Stuff that matters.

A comment on Ask HN: Has anyone ever crawled over a billion pages? How much did it cost?. Parent comment

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by mtmail on Tuesday September 13, 2016 @06:41PM (#12492843)
Deusu's crawl servers are located at https://www.hosteurope.de/en/Server/Root-Server/ while the website points to his home broadband ISP. Two servers at his specs would be 200 Euro/month total, with 5x more bandwidth than he currently uses. I'd say that's much cheaper that AWS. Of course crawl companies charge more: they run a business, pay system administrators, have more backup and redundancy.
[ Parent | on HN ]
Re:Ask HN: Has anyone ever crawled over a billion pages? How... by outpan on Tuesday September 13, 2016 @06:48PM (#12492873)
I'm not sure how he manages to crawl with this speed using such low amount of resources.

We did a benchmark on Nutch and couldn't really pass the 10-14 M(B)ps on a $1200/month machine. Even though we hired a professional to optimize the setup. The same is roughly true about Heritrix.

Just wondering if there is something missing in his setup, such as domain/ip rate limiting.

[ Parent | on HN ]
Re:Ask HN: Has anyone ever crawled over a billion pages? How... by detaro on Tuesday September 13, 2016 @06:50PM (#12492883)
You can check his source if you are curious how it works ;)
[ Parent | on HN ]