News for hackers. Stuff that matters.
Re:The joys and sorrows of maintaining a personal website by franze on Saturday May 07, 2022 @04:04AM (#31292922)
Googlebot supports their set of robots.txt instructions https://developers.google.com/search/docs/advanced/robots/cr...

Crawl Delay is not part of it.

There are no comments on this comment yet.