News for hackers. Stuff that matters.
Re:Google to Explore Alternatives to Robots.txt by judge2020 on Saturday July 08, 2023 @12:12PM (#36645755)
You can block the crawler on your entire site. I’m not sure it’s true that it’s primarily used “to avoid overloading your site”.
Re:Google to Explore Alternatives to Robots.txt by blacksmith_tb on Saturday July 08, 2023 @01:18PM (#36646453)
For sure, since those directives in your robots.txt don't actually compel the crawlers to do anything. They're more like a polite request, and plenty of bots ignore or 'accidentally' overstep them. I do think they have still some value, not just as a handy list of high-value targets - you may know that some part of your site has a bunch of similar links that it doesn't make sense to crawl or index (though there's always norel/nofollow...), or that some pages (/account/preferences etc.) just don't make sense for bots to be visiting. The general idea of extending the standard to cover training AI isn't a terrible idea, but it does seem like too little, too late.
Re:Google to Explore Alternatives to Robots.txt by lakomen on Saturday July 08, 2023 @09:35PM (#36650613)
robots.txt tells the search engine which content is relevant. That's all.