Re:The joys and sorrows of maintaining a personal website
by sorokod on Saturday May 07, 2022 @04:12AM
(#31292954)
I know it amounts to same, but from the users perspective it would be more accurate to say that Google doesn't respect crawl delay.
[ Parent
| on HN ]
Re:The joys and sorrows of maintaining a personal website
by franze on Saturday May 07, 2022 @05:38AM
(#31293233)
crawl delay was an extension of /robots.txt and never a part of the original "recommendation"
and also never made it into the standard draft https://datatracker.ietf.org/doc/html/draft-rep-wg-topic-00#...