Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Web Scraping a Javascript Heavy Website: Keeping Things Simple.
1. Documented API. Failing that...
2. HTTP client fetching structured data (XHR calls). Failing that...
3. HTTP client fetching and scraping HTML documents. Failing that...
4. Headless browser
I recently found myself pushed to #4 to handle sites with over-complex JS or anti-automation techniques.
There are no comments on this comment yet.
1. Documented API. Failing that...
2. HTTP client fetching structured data (XHR calls). Failing that...
3. HTTP client fetching and scraping HTML documents. Failing that...
4. Headless browser
I recently found myself pushed to #4 to handle sites with over-complex JS or anti-automation techniques.