Re:Show HN: CloudScrape – Cloud-based web scraping platform
by manigandham on Sunday October 18, 2015 @08:13AM
(#10408004)
> If you don't want someone to access your page, then don't respond to their request
> there's no easy way to always reliably identify the requester
That's the problem: you can't identify the person to block them in the first place.
Robots.txt is actually an explicit signal of intention for reputable search engines but that's all we have today and is easily ignored and does not work with these scrapers or anyone else.
Not sure what your last sentence means.
Even if identification was hard, which is not true because of how HTTP works, it is irrelevant because HTTP doesn't discriminate. If someone does, that is their problem, and should be solved by them, and not a committee or law.