News for hackers. Stuff that matters.
Re:Show HN: Scrape the web by injecting JavaScript into web ... by MoOmer on Tuesday December 23, 2014 @09:20AM (#8788168)
Are you rendering the JavaScript on those pages? What's your in-domain delay?
Re:Show HN: Scrape the web by injecting JavaScript into web ... by jalfresi on Tuesday December 23, 2014 @09:45AM (#8788268)
Yes, and injecting JS into the page for easy analysis/collection. The delay is dynamic, based on captcha rates, proxy load and historic captcha costs. It constantly checks the current running costs and throttles the number of requests over the hour.