News for hackers. Stuff that matters.
Re:The joys and sorrows of maintaining a personal website by freedomben on Friday May 06, 2022 @10:20PM (#31291632)
From what I've been told, the Google Bot absolutely tries to respect the robots.txt. The advice I got was to double check that my syntax and expressions were really correct. A lot of people screw it up so it has to be basically perfect otherwise it won't be parsed correctly. Are you willing to share your robots.txt file?
Re:The joys and sorrows of maintaining a personal website by franze on Saturday May 07, 2022 @04:04AM (#31292922)
Googlebot supports their set of robots.txt instructions https://developers.google.com/search/docs/advanced/robots/cr...

Crawl Delay is not part of it.