News for hackers. Stuff that matters.
Re:Content Independence Day: no AI crawl without compensation by techjamie on Wednesday July 02, 2025 @04:56PM (#44448720)
Many of these tarpits deliberately serve the data at an excruciatingly low speed to ease the burden on the server resources. It's cheaper than quickly serving the same crawlers your entire website at max speed constantly.
Re:Content Independence Day: no AI crawl without compensation by OutOfHere on Wednesday July 02, 2025 @08:12PM (#44450221)
If we are going for cheaper, how is it cheaper than an HTTP 429 error? It's not.
Re:Content Independence Day: no AI crawl without compensation by DamonHD on Thursday July 03, 2025 @03:53AM (#44452666)
Virtually nothing pays attention to 429s. More things pay attention to 500s and 503s. Some however use those as a trigger to repoll immediately.
Re:Content Independence Day: no AI crawl without compensation by felurx on Sunday July 06, 2025 @03:20PM (#44483244)
I suppose they're going for a trade-off between "cheaper" and "satisfying" (as in, the satiafaction of sticking it to unethical shitty companies)