Re:Google to Explore Alternatives to Robots.txt
by dasil003 on Saturday July 08, 2023 @12:26PM
(#36645918)
I don't understand what you have against robots.txt. It's just a way to signal what you want crawlers to do on your site. It's not complicated or mandatory, but it gives you a way to influence how your site is accessed. I'm not sure why you would jump straight to litigation as a better solution—that solves a much smaller set of problems at a much higher cost.
[ Parent
| on HN ]
There are no comments on this comment yet.