RoboBrowser: Your friendly neighborhood web scraper

Re:RoboBrowser: Your friendly neighborhood web scraper by Benfromparis on Thursday May 12, 2016 @08:16AM (#11682792)

Interesting for unprotected websites but it's easy to detect and to block: no valid js, no valid meta header, no valid cookie, suspect behavior...

Selenium is a much "elaborated" solution, but still, can be detected most of the time.

Disclosure: I'm DataDome co-founder. If you want to detect bad bots and scrapers on your website, don't hesitate to try out for free and to share your feedback with us https://datadome.co

[ Parent | on HN ]

Re:RoboBrowser: Your friendly neighborhood web scraper by dchuk on Thursday May 12, 2016 @02:27PM (#11685630)

I realize you have reasons not to answer this question, but out of curiosity, what sorts of thing can tip off the fact that a site is getting scraped by a real browser and selenium?

[ Parent | on HN ]

Re:RoboBrowser: Your friendly neighborhood web scraper by Benfromparis on Friday May 13, 2016 @04:26AM (#11689291)

Of course I cannot go much into details, but we are using behavior detection and Javascript tracking (mouse, scroll, screen...).

[ Parent | on HN ]