Web Scraping: Bypassing “403 Forbidden,” captchas, and more

Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by retube on Thursday March 16, 2017 @10:35AM (#13885309)

so can you use greasemonkey to follow links, load new page, parse new page, just like a headless crawler?

[ Parent | on HN ]

Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by m00dy on Thursday March 16, 2017 @11:11AM (#13885638)

I think you can use a 3rd party backend service to store state of the crawler. So, when page reloads, you know which state you are in.

[ Parent | on HN ]

Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by retube on Thursday March 16, 2017 @11:35AM (#13885866)

right. so basically a greasemonkey script is scoped to the current page? Is there any scripting solution that is not scoped to current page? In chrome maybe?

[ Parent | on HN ]

Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by cryptarch on Thursday March 16, 2017 @11:39AM (#13885911)

Browser automation via (realistically Seleniun) WebDriver or a proxy that inserts scripts (like TestCafe).

[ Parent | on HN ]

Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by tehlike on Thursday March 16, 2017 @12:25PM (#13886317)

selenium with webdriver(io) or phantomjs. I like selenium more, though.

[ Parent | on HN ]