News for hackers. Stuff that matters.

A comment on Web Scraping: Bypassing “403 Forbidden,” captchas, and more. Parent comment

Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by elchief on Thursday March 16, 2017 @01:57PM (#13887244)
We use HTMLUnit. Works pretty well. Not super fast, but you want to scrape individual sites at a moderate rate anyway
[ Parent | on HN ]
Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by skinnymuch on Thursday March 16, 2017 @08:06PM (#13890270)
Have you run into issues? I'd think HTMLUnit isn't robust enough and it's "browser" Ian limiting?
[ Parent | on HN ]
Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by elchief on Tuesday March 21, 2017 @12:04AM (#13919793)
It's got a couple of idiosyncrasies but works well in general. Barfs out too much log info in general. XPath 1 is limiting, but can use Saxon if need to
[ Parent | on HN ]
Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more by skinnymuch on Saturday March 25, 2017 @10:33AM (#13955832)
Holy crap! Xpath 1 is still being used for it? I actually have no clue what the differences are between xpath versions but I just assumed everyone is on xpath 2.

I guess my other question is - have you run into any situations where the JavaScript parsing or browser rendering wasn't good enough?

[ Parent | on HN ]