Ask HN: How are you dealing with scraping hits from EC2 machines?

Re:Ask HN: How are you dealing with scraping hits from EC2 m... by chewxy on Wednesday December 12, 2012 @09:10AM (#4910068)

Let's examine your motivation: why do you want to block said scrapers in the first place? SEO concerns (dupe content)?

[ Parent | on HN ]

Re:Ask HN: How are you dealing with scraping hits from EC2 m... by digitalpbk on Wednesday December 12, 2012 @09:17AM (#4910090)

Mostly duplicate content & messing up my analytics (increased bounce rate, decreased time spend on page etc.)

[ Parent | on HN ]

Re:Ask HN: How are you dealing with scraping hits from EC2 m... by digitalpbk on Wednesday December 12, 2012 @09:33AM (#4910197)

Because it seems to be from selenium (from referer), it is triggering the JS too, we are using Google Analytics.

[ Parent | on HN ]

Re:Ask HN: How are you dealing with scraping hits from EC2 m... by ig1 on Wednesday December 12, 2012 @09:44AM (#4910236)

Why not exclude the EC2 ip range from analytics ?

[ Parent | on HN ]

Re:Ask HN: How are you dealing with scraping hits from EC2 m... by dumbfounder on Wednesday December 12, 2012 @09:29AM (#4910166)

What analytics are you using? I thought most would use javascript to avoid problems like this, and I would wager that the vast majority of bots don't bother executing javascript. You will always have legit bots hitting your site as well.

[ Parent | on HN ]

Re:Ask HN: How are you dealing with scraping hits from EC2 m... by bdcravens on Wednesday December 12, 2012 @09:48AM (#4910257)

Selenium executes Javascript, as do other WebKit-based scrapers, like phantom.js.

[ Parent | on HN ]