News for hackers. Stuff that matters.
Re:Web Search API by sreekanth850 on Monday October 05, 2026 @10:34AM (#49965438)
And you think all this web AI crawlers will respect robots.txt. That era is gone.
Re:Web Search API by Jskewel on Monday October 05, 2026 @03:00PM (#49968981)
The big USA ones do, and it would be madness for them to do otherwise.

But that's meaningless because 99% of AI crawlers are "bad bots" which ignore robots.txt and use domestic IPs to circumvent blocks.

Re:Web Search API by dawidpotocki on Monday October 05, 2026 @03:30PM (#49969371)
That doesn't match my experience. For example, I recently had Meta scrape robots.txt disallowed paths. On some page they claimed they respect it but they don't. And yes, all the IPs they used to connect to me (they used a different /64 network for each connection) were owned by Meta, it was not someone pretending to be them.

Some other reports of this: https://github.com/TecharoHQ/anubis/issues/1565

I ended up just fully closing connections with no response from these assholes on any URL.

Re:Web Search API by mitxela on Tuesday October 06, 2026 @07:53AM (#49977129)
The era of robots.txt files being reasonable is gone too. Usually they either block everything, or block everything except Googlebot. If you want to make a search engine at all, you have to ignore robots.txt, ignore noindex, ignore nofollow, and so on. If we had your way, Google would have a legally enforced search engine monopoly.