News for hackers. Stuff that matters.

A comment on Build a crawler to crawl million pages with only one machine in just 2 hours. Parent comment

Re:Build a crawler to crawl million pages with only one mach... by chrisabrams on Tuesday February 28, 2017 @11:54AM (#13755125)
Oh boy. When I lived in San Francisco the Python community had this on their coffee mugs :O
[ Parent | on HN ]
Re:Build a crawler to crawl million pages with only one mach... by dickbasedregex on Wednesday March 01, 2017 @01:27PM (#13766658)
Your bottleneck should probably be managing your requests, not saving the document/assets. I'm dismissive that language speed is non-negligible in scrapping. Rate limiting and being smart about how you're fetching data should probably be your concern. ... sure, if you don't mind slamming a server with concurrent requests, your language choice might start to matter if your IP isn't blocked first.
[ Parent | on HN ]