News for hackers. Stuff that matters.
Re:Google to Explore Alternatives to Robots.txt by lokar on Saturday July 08, 2023 @10:48AM (#36644941)
Google just copies everything and saves it. They then use it for various uses. It would be strange to fetch pages several times.
Re:Google to Explore Alternatives to Robots.txt by marginalia_nu on Saturday July 08, 2023 @11:37AM (#36645399)
They don't really need to fetch it twice though? Fetch 0-1 times, use it according to what robots.txt allows.