Re:Robots.txt meant for search engines don’t work well for w...
by eknkc on Friday April 21, 2017 @03:49AM
(#14163771)
I don't think a poorly written scraper would follow robots.txt rules according to spec. So, in any case the site should have other measures (rate limiting?) anyway.
[ Parent
| on HN ]
There are no comments on this comment yet.