Re:Google to Explore Alternatives to Robots.txt
by marginalia_nu on Saturday July 08, 2023 @11:37AM
(#36645399)
They don't really need to fetch it twice though? Fetch 0-1 times, use it according to what robots.txt allows.
[ Parent
| on HN ]
There are no comments on this comment yet.