News for hackers. Stuff that matters.
Re:How to scrape anything on the web and not get caught by Theodores on Sunday April 22, 2018 @03:50AM (#16895173)
I have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get pages serving quickly. Somehow I missed the memo on how to set your site up to not be scraped. Is there an nginx setting for that?

This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond looking at logs and doing IP bans?

Re:How to scrape anything on the web and not get caught by pjc50 on Sunday April 22, 2018 @03:59AM (#16895193)
The cheap technique is rate limiting, but there are also "canary pages" (linked invisibly but should not be accessed by humans) and other techniques for looking to see if the flow through the site is as expected.
Re:How to scrape anything on the web and not get caught by jstarfish on Sunday April 22, 2018 @04:35AM (#16895286)
IPs are naive and too broad. Force users to authenticate to see your content. Unauthenticated viewers get to see only a scrambled version. Now you can track usage by username.

Establish a baseline as to what constitutes "normal" browsing.

Any user exceeding a threshold of page requests gets rate limited/banned.

Stopping people from recording public information is unrealistic, but you can certainly make it more of a pain in the ass.

Re:How to scrape anything on the web and not get caught by mmcwilliams on Sunday April 22, 2018 @08:09AM (#16895826)
I'm sort of on the other side of this problem, but I think an interesting counter-measure came up in a recent Show HN[0].

IP bans alone are probably not enough to stop a motivated scraper, or work longterm, so checking whether or not traffic is coming from a cloud provider and throwing up a captcha would be a significant hurdle, while allowing humans on cloud-hosted VPNs to pass through without much trouble.

[0] https://news.ycombinator.com/item?id=16868012

Re:How to scrape anything on the web and not get caught by amelius on Sunday April 22, 2018 @06:25AM (#16895542)
Captchas are one (drastic) option.
Re:How to scrape anything on the web and not get caught by nojvek on Sunday April 22, 2018 @11:43AM (#16896631)
Any image based captcha where one needs to identify words from an image can be easily broken by algorithms now.

Google would be the leader with re-captcha, but as a human, I fail a whole number of them. They are a very annoying experience to your users.