News for hackers. Stuff that matters.
Re:Robots.txt meant for search engines don’t work well for w... by timClicks on Friday April 21, 2017 @03:38AM (#14163726)
The Crawl-delay directive is the de facto standard for this.

There are no comments on this comment yet.