News for hackers. Stuff that matters.

A comment on AI crawlers haven't learned to play nice with websites. Parent comment

Re:AI crawlers haven't learned to play nice with websites by bayindirh on Tuesday March 18, 2025 @10:10AM (#43399632)
It's already well known that none of the search engines, crawlers, and companies honor robots.txt (anymore).

It's just for us oldies who prefer the ethical web where people respected the RFCs they have collectively written.

[ Parent | on HN ]
Re:AI crawlers haven't learned to play nice with websites by pimlottc on Tuesday March 18, 2025 @10:23AM (#43399806)
It’s one thing to know they don’t honor it, it’s another to know they intentionally abuse it
[ Parent | on HN ]
Re:AI crawlers haven't learned to play nice with websites by johnea on Wednesday March 19, 2025 @08:03PM (#43418592)
Yes, it is, and now we know both of those things don't we.

What will be the consequence? My risky prediction: Absolutely Nothing, for neither goggle nor any of the other miscreants running the bot nets.

And just to take a poke at the title of the original El Reg article: Do you really think the problem is that they just haven't "learned" yet how to "play nice"? I'm sure the great oracle goggle, just needs someone to poperly splain it to them.

[ Parent | on HN ]