Re:Google to Explore Alternatives to Robots.txt
by lokar on Saturday July 08, 2023 @10:48AM
(#36644941)
Google just copies everything and saves it. They then use it for various uses. It would be strange to fetch pages several times.
[ Parent
| on HN ]
Re:Google to Explore Alternatives to Robots.txt
by marginalia_nu on Saturday July 08, 2023 @11:37AM
(#36645399)
They don't really need to fetch it twice though? Fetch 0-1 times, use it according to what robots.txt allows.