News for hackers. Stuff that matters.
Re:Show HN: Scrape the web by injecting JavaScript into web ... by TheEspion on Tuesday December 23, 2014 @08:59AM (#8788102)
The packages are for a set number of pages, with no timeframe. I don't have prices yet for a load in the millions of pages per hour. What kind of system do you feed data into?
Re:Show HN: Scrape the web by injecting JavaScript into web ... by jalfresi on Tuesday December 23, 2014 @09:46AM (#8788275)
A very large array of mysql databases. Basically, each month we fire up a new fresh database and start streaming data into it. Were currently pulling around 700Gb a month. Our reporting tools/systems run queries across this array. Its actually not that bad speed wise (reports of over 9000 keywords over a 1 week period for top 100 positions on a per hour basis)