News for hackers. Stuff that matters.
Re:Ask HN: Get feedback on my bootstrapped web scraping as a... by Jake232 on Wednesday September 24, 2014 @08:07PM (#8364851)
Needs more details, a couple of examples that instantly spring to mind:

Do you spread requests out across multiple IP's to avoid bans / rate limiting?

Do you run javascript on the pages?

Re:Ask HN: Get feedback on my bootstrapped web scraping as a... by notastartup on Wednesday September 24, 2014 @10:15PM (#8365209)
No, it will not fly through thousands of pages at a blinding rate using multiple ip addresses, it goes about at a human like pace to avoid getting banned. If an ip is banned a new one may be assigned. If you need to go faster you will need concurrent jobs.

All the jobs use chrome browser, and supports any website.