Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr...
by bdcravens on Sunday December 09, 2012 @01:10AM
(#4893997)
See my other comment :-)
I've been doing this on a site that is 100% Javascript-driven for over a year, very successfully.
It's really no different than hitting a static site with Selenium. Figuring out the proper XPath to use is often the biggest challenge: Chrome Developer tools help immensely. Also, you need to watch for delays in JS rendering, so put a lot of pauses in your scripts.
It's of course slow, so if you want to distribute it across several machines, use Selenium Grid or a queue system (SQS, Resque, etc). Setup Xvfb to run on headless Linux instances.
I've been doing this on a site that is 100% Javascript-driven for over a year, very successfully.
It's really no different than hitting a static site with Selenium. Figuring out the proper XPath to use is often the biggest challenge: Chrome Developer tools help immensely. Also, you need to watch for delays in JS rendering, so put a lot of pauses in your scripts.
It's of course slow, so if you want to distribute it across several machines, use Selenium Grid or a queue system (SQS, Resque, etc). Setup Xvfb to run on headless Linux instances.