Re:How to scrape anything on the web and not get caught
by pjc50 on Sunday April 22, 2018 @03:59AM
(#16895193)
The cheap technique is rate limiting, but there are also "canary pages" (linked invisibly but should not be accessed by humans) and other techniques for looking to see if the flow through the site is as expected.
Re:How to scrape anything on the web and not get caught
by jstarfish on Sunday April 22, 2018 @04:35AM
(#16895286)
IPs are naive and too broad. Force users to authenticate to see your content. Unauthenticated viewers get to see only a scrambled version. Now you can track usage by username.
Establish a baseline as to what constitutes "normal" browsing.
Any user exceeding a threshold of page requests gets rate limited/banned.
Stopping people from recording public information is unrealistic, but you can certainly make it more of a pain in the ass.
Re:How to scrape anything on the web and not get caught
by mmcwilliams on Sunday April 22, 2018 @08:09AM
(#16895826)
I'm sort of on the other side of this problem, but I think an interesting counter-measure came up in a recent Show HN[0].
IP bans alone are probably not enough to stop a motivated scraper, or work longterm, so checking whether or not traffic is coming from a cloud provider and throwing up a captcha would be a significant hurdle, while allowing humans on cloud-hosted VPNs to pass through without much trouble.
Re:How to scrape anything on the web and not get caught
by amelius on Sunday April 22, 2018 @06:25AM
(#16895542)
Captchas are one (drastic) option.
Re:How to scrape anything on the web and not get caught
by nojvek on Sunday April 22, 2018 @11:43AM
(#16896631)
Any image based captcha where one needs to identify words from an image can be easily broken by algorithms now.
Google would be the leader with re-captcha, but as a human, I fail a whole number of them. They are a very annoying experience to your users.
This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond looking at logs and doing IP bans?