News for hackers. Stuff that matters.

A comment on I protect my forgejo instance from AI Web Crawlers. Parent comment

Re:I protect my forgejo instance from AI Web Crawlers by andai on Monday December 22, 2025 @09:33AM (#46354405)
No I meant like, if you have a blog with 10 posts.. do they just scrape the same 10 pages thousands of times?

Because people are reporting constant traffic, which would imply that the site is being scraped millions of times per year. How does that make any sense? Are there millions of AI companies?

[ Parent | on HN ]
Re:I protect my forgejo instance from AI Web Crawlers by marcthe12 on Monday December 22, 2025 @10:10AM (#46354713)
Basically the scrappers do not bother to cache your website or if they do, with an insanely low ttl. Also they do not specialize the content. So the worst hit sites are something like git hosting due the bfs style scrape (every link). The worst part is alot of this is done via tunneling so ip can be different each time or from residential ops. Which makes it annoying.
[ Parent | on HN ]