News for hackers. Stuff that matters.
Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by jmkni on Thursday March 16, 2017 @12:40PM (#13886445)
I have used Selenium for this with quite a bit of success, or as others have mentioned, just figure out where the API endpoints are with fiddler and pull the data directly from the source.

Sometimes this can be a PITA though, for example Tableau obfuscates the JSON they send back, so it's easier to use Selenium to wait ten seconds and then scrape the resulting HTML.

There are no comments on this comment yet.