Ask HN: Should you block web scrapers?

Re:Ask HN: Should you block web scrapers? by ldjb on Monday February 01, 2016 @01:54PM (#11014255)

There is no way to completely block scrapers. The most you can do is make life difficult for those writing them. You also have to ensure you're not blocking genuine users.

Instead of investing time and money trying to prevent competitors, why not use that time and money to innovate on the service you provide so that there will be no point in setting up a competitor?

[ Parent | on HN ]

Re:Ask HN: Should you block web scrapers? by jorgecurio on Monday February 01, 2016 @03:39PM (#11015214)

Of course it's not possible to 100% block ALL scrapers, much like the debate surrounding obfusticating javascript.

The point is to increase the cost of developing a scraper to a maxima where ROI begins to rapidly deteriorate. I'm trying to identify techniques and methods which will:

- identify real users apart from bot users

- increase the costs for the bot

[ Parent | on HN ]