News for hackers. Stuff that matters.
Re:Google changes its search box by deaton on Tuesday May 19, 2026 @03:42PM (#48198391)
What you gain? Nothing, but they and other AI companies have decided not to respect your robots.txt
Re:Google changes its search box by loloquwowndueo on Tuesday May 19, 2026 @04:02PM (#48198677)
There are other ways to block robots from crawling our sites. I have a robots.txt but place no faith in it, it’s just there because it’s cheap and does stop some of the crawlers.