News for hackers. Stuff that matters.
Re:Content Independence Day: no AI crawl without compensation by agentultra on Wednesday July 02, 2025 @01:19PM (#44446318)
A nice attempt and another layer for the swiss cheese of technology it will take to try and ease the burden AI companies are putting on people trying to run websites.

I'd be cautious about relying on just the good will of Cloudflare.

It's unfortunate that we need honeypots and tarpits to trap AI scrapers just so that our hosting bills don't get hosed. It's taking a good chunk of value out of running a site on the Internet.

Re:Content Independence Day: no AI crawl without compensation by OutOfHere on Wednesday July 02, 2025 @01:33PM (#44446513)
Feel free to waste your expensive outgoing bandwidth running malware. It is a genius idea really from the cloud companies to enrich their balances.

Definitely don't rewrite your web server more efficiently in Rust instead. /s

Re:Content Independence Day: no AI crawl without compensation by Retric on Wednesday July 02, 2025 @02:16PM (#44447068)
Serving poisoned text can be so cheap it’s effectively free as long as you don’t give them a lot of links.
Re:Content Independence Day: no AI crawl without compensation by Mars008 on Wednesday July 02, 2025 @08:22PM (#44450287)
Yeh, and say goodby to google search. You didn't want to be there anyway, right?
Re:Content Independence Day: no AI crawl without compensation by Retric on Thursday July 03, 2025 @06:33AM (#44453582)
Google makes it easy to identify their bot. Often people want to do this to give them more access.

People care about AI companies because they’re ignoring robots.txt etc.

Re:Content Independence Day: no AI crawl without compensation by OutOfHere on Thursday July 03, 2025 @10:40AM (#44455555)
Another thing that doesn't make sense is why it has to be poisoned text. Why can't it just be a mix of whitespace? I doubt anyone is using LLMs with streaming inputs to determine whether to continue reading the page.
Re:Content Independence Day: no AI crawl without compensation by Retric on Thursday July 03, 2025 @11:25AM (#44456023)
Company’s actively harming you should be discouraged, preferably by running them out of business. Whitespace doesn’t do that and makes it easy to identify when the crawlers fail.

Swapping meaning poisons the LLM but makes it really difficult for a preprocessing step to understand the difference between good and bad inputs.

Re:Content Independence Day: no AI crawl without compensation by techjamie on Wednesday July 02, 2025 @04:56PM (#44448720)
Many of these tarpits deliberately serve the data at an excruciatingly low speed to ease the burden on the server resources. It's cheaper than quickly serving the same crawlers your entire website at max speed constantly.
Re:Content Independence Day: no AI crawl without compensation by OutOfHere on Wednesday July 02, 2025 @08:12PM (#44450221)
If we are going for cheaper, how is it cheaper than an HTTP 429 error? It's not.
Re:Content Independence Day: no AI crawl without compensation by DamonHD on Thursday July 03, 2025 @03:53AM (#44452666)
Virtually nothing pays attention to 429s. More things pay attention to 500s and 503s. Some however use those as a trigger to repoll immediately.
Re:Content Independence Day: no AI crawl without compensation by felurx on Sunday July 06, 2025 @03:20PM (#44483244)
I suppose they're going for a trade-off between "cheaper" and "satisfying" (as in, the satiafaction of sticking it to unethical shitty companies)