News for hackers. Stuff that matters.
Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by DrFalkyn on Tuesday March 25, 2025 @10:29PM (#43478246)
Why kind of data that isn’t public would be so valuable for AI training?

Seems like there’s a fuck ton. All of Wikipedia, GitHub for code, etc.

I can understand targeting certain sites like Reddit, etc. but not random websites

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by timewizard on Tuesday March 25, 2025 @10:44PM (#43478338)
It's to rip off copyrighted content and profit from it instead of the original authors. It's like every other low rent and highly automated scam that finds it's way onto the internet.

If you look closely even Google does this. This is probably why many popular sites started getting down ranked in the last 2 years. Now they're below the fold and Google can present their content as their own through the AI box.

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by throwaway2037 on Tuesday March 25, 2025 @11:01PM (#43478430)
Please remember that Google only needs to be marginally better than the competition. And, of course, their primary biz is ads, not serving great results; that is a distant second priority.
Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by MathMonkeyMan on Wednesday March 26, 2025 @01:35AM (#43479163)
Their biz is ads, but since search is winner takes all they need only be marginally better than the competition... twenty years ago.
Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by timewizard on Wednesday March 26, 2025 @02:39AM (#43479425)
> Their biz is ads,

Yea, but, the FTC doesn't want it to be.

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by grotorea on Wednesday March 26, 2025 @01:51PM (#43484820)
Discord I guess would be quite valuable, even the de facto public servers.