News for hackers. Stuff that matters.
Re:Robots.txt meant for search engines don’t work well for w... by eknkc on Friday April 21, 2017 @03:49AM (#14163771)
I don't think a poorly written scraper would follow robots.txt rules according to spec. So, in any case the site should have other measures (rate limiting?) anyway.

There are no comments on this comment yet.