News for hackers. Stuff that matters.
Re:Google to Explore Alternatives to Robots.txt by marginalia_nu on Saturday July 08, 2023 @11:37AM (#36645399)
They don't really need to fetch it twice though? Fetch 0-1 times, use it according to what robots.txt allows.

There are no comments on this comment yet.