News for hackers. Stuff that matters.
Re:I was wrong about robots.txt by edoceo on Thursday July 17, 2025 @12:14AM (#44589653)
Or getting a CAPTCHA from Chrome when visiting a site you've been to dozens of times (Stack Overflow). Now I just skip that content, probably in my LLM already anyway.
Re:I was wrong about robots.txt by realusername on Thursday July 17, 2025 @01:27AM (#44589972)
It's the same thing as the anti pirate ads, you only annoy legit customers, this agressive captcha campaign just makes Stackoverflow drop down even faster than it would normally by making it lower quality.
Re:I was wrong about robots.txt by codingminds on Thursday July 17, 2025 @01:45AM (#44590080)
Keep in mind that those LLMs are one of the bigger reasons why we see more and more anti bot behaviour on sites like SO.

That aggressive crawling to train those on everything is insane.