News for hackers. Stuff that matters.
Re:Web Scraping a Javascript Heavy Website: Keeping Things S... by timscott on Thursday August 29, 2013 @12:53AM (#6294337)
I've recently been learning all this the hard way.

1. Documented API. Failing that...

2. HTTP client fetching structured data (XHR calls). Failing that...

3. HTTP client fetching and scraping HTML documents. Failing that...

4. Headless browser

I recently found myself pushed to #4 to handle sites with over-complex JS or anti-automation techniques.

There are no comments on this comment yet.