Re:Robots.txt meant for search engines don’t work well for w...
by timClicks on Friday April 21, 2017 @03:38AM
(#14163726)
The Crawl-delay directive is the de facto standard for this.
[ Parent
| on HN ]
There are no comments on this comment yet.