Re:Content Independence Day: no AI crawl without compensation
by OutOfHere on Wednesday July 02, 2025 @01:33PM
(#44446513)
Feel free to waste your expensive outgoing bandwidth running malware. It is a genius idea really from the cloud companies to enrich their balances.
Definitely don't rewrite your web server more efficiently in Rust instead. /s
Re:Content Independence Day: no AI crawl without compensation
by Retric on Wednesday July 02, 2025 @02:16PM
(#44447068)
Serving poisoned text can be so cheap it’s effectively free as long as you don’t give them a lot of links.
Re:Content Independence Day: no AI crawl without compensation
by Mars008 on Wednesday July 02, 2025 @08:22PM
(#44450287)
Yeh, and say goodby to google search. You didn't want to be there anyway, right?
Re:Content Independence Day: no AI crawl without compensation
by Retric on Thursday July 03, 2025 @06:33AM
(#44453582)
Google makes it easy to identify their bot. Often people want to do this to give them more access.
People care about AI companies because they’re ignoring robots.txt etc.
Re:Content Independence Day: no AI crawl without compensation
by OutOfHere on Thursday July 03, 2025 @10:40AM
(#44455555)
Another thing that doesn't make sense is why it has to be poisoned text. Why can't it just be a mix of whitespace? I doubt anyone is using LLMs with streaming inputs to determine whether to continue reading the page.
Re:Content Independence Day: no AI crawl without compensation
by Retric on Thursday July 03, 2025 @11:25AM
(#44456023)
Company’s actively harming you should be discouraged, preferably by running them out of business. Whitespace doesn’t do that and makes it easy to identify when the crawlers fail.
Swapping meaning poisons the LLM but makes it really difficult for a preprocessing step to understand the difference between good and bad inputs.
Re:Content Independence Day: no AI crawl without compensation
by techjamie on Wednesday July 02, 2025 @04:56PM
(#44448720)
Many of these tarpits deliberately serve the data at an excruciatingly low speed to ease the burden on the server resources. It's cheaper than quickly serving the same crawlers your entire website at max speed constantly.
Re:Content Independence Day: no AI crawl without compensation
by OutOfHere on Wednesday July 02, 2025 @08:12PM
(#44450221)
If we are going for cheaper, how is it cheaper than an HTTP 429 error? It's not.
I'd be cautious about relying on just the good will of Cloudflare.
It's unfortunate that we need honeypots and tarpits to trap AI scrapers just so that our hosting bills don't get hosed. It's taking a good chunk of value out of running a site on the Internet.