News for hackers. Stuff that matters.
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr... by bdcravens on Sunday December 09, 2012 @01:10AM (#4893997)
See my other comment :-)

I've been doing this on a site that is 100% Javascript-driven for over a year, very successfully.

It's really no different than hitting a static site with Selenium. Figuring out the proper XPath to use is often the biggest challenge: Chrome Developer tools help immensely. Also, you need to watch for delays in JS rendering, so put a lot of pauses in your scripts.

It's of course slow, so if you want to distribute it across several machines, use Selenium Grid or a queue system (SQS, Resque, etc). Setup Xvfb to run on headless Linux instances.

There are no comments on this comment yet.