Re:Show HN: Scrape the web by injecting JavaScript into web ...
by jalfresi on Tuesday December 23, 2014 @08:43AM
(#8788054)
The packages don't state if the number of pages are per month, day or hour? We currently scrape well over 5 million pages an hour for a lot less (although, much like you we are geared up for such loads) but it would be interesting to see the cost per number of pages per hour you charge for odd-jobs/one-offs.
[ Parent
| on HN ]
Re:Show HN: Scrape the web by injecting JavaScript into web ...
by TheEspion on Tuesday December 23, 2014 @08:59AM
(#8788102)
The packages are for a set number of pages, with no timeframe. I don't have prices yet for a load in the millions of pages per hour. What kind of system do you feed data into?
Re:Show HN: Scrape the web by injecting JavaScript into web ...
by jalfresi on Tuesday December 23, 2014 @09:46AM
(#8788275)
A very large array of mysql databases. Basically, each month we fire up a new fresh database and start streaming data into it. Were currently pulling around 700Gb a month. Our reporting tools/systems run queries across this array. Its actually not that bad speed wise (reports of over 9000 keywords over a 1 week period for top 100 positions on a per hour basis)
Re:Show HN: Scrape the web by injecting JavaScript into web ...
by willu on Tuesday December 23, 2014 @09:35AM
(#8788224)
If you are able to crawl that volume as cheaply as you say, you should definitely be offering it as a service.
Re:Show HN: Scrape the web by injecting JavaScript into web ...
by jalfresi on Tuesday December 23, 2014 @09:43AM
(#8788258)
Its a very single purpose system, so probably not amenable to general purpose crawling (though we do have a separate system that is based of the design of our core system that is a general purpose web crawler/indexer)
Re:Show HN: Scrape the web by injecting JavaScript into web ...
by MoOmer on Tuesday December 23, 2014 @09:20AM
(#8788168)
Are you rendering the JavaScript on those pages? What's your in-domain delay?
Re:Show HN: Scrape the web by injecting JavaScript into web ...
by jalfresi on Tuesday December 23, 2014 @09:45AM
(#8788268)
Yes, and injecting JS into the page for easy analysis/collection. The delay is dynamic, based on captcha rates, proxy load and historic captcha costs. It constantly checks the current running costs and throttles the number of requests over the hour.
Re:Show HN: Scrape the web by injecting JavaScript into web ...
by bolasanibk on Tuesday December 23, 2014 @12:59PM
(#8789211)
Is there any place I can read more about the technical side of this. I would love to know how you achieve these rates.