News for hackers. Stuff that matters.
Re:The joys and sorrows of maintaining a personal website by franze on Saturday May 07, 2022 @04:01AM (#31292909)
Google does not support crawl delay.

You can set your crawl rate lower every 90 days here https://www.google.com/webmasters/tools/settings

other possibilities are here https://support.google.com/webmasters/answer/48620?hl=en there is a way to lower crawling without total blocking (returning HTTP 503 if there is too much load)

Re:The joys and sorrows of maintaining a personal website by sorokod on Saturday May 07, 2022 @04:12AM (#31292954)
I know it amounts to same, but from the users perspective it would be more accurate to say that Google doesn't respect crawl delay.
Re:The joys and sorrows of maintaining a personal website by franze on Saturday May 07, 2022 @05:38AM (#31293233)
crawl delay was an extension of /robots.txt and never a part of the original "recommendation"

and also never made it into the standard draft https://datatracker.ietf.org/doc/html/draft-rep-wg-topic-00#...