Block AI bots, scrapers and crawlers with a single click

Re:Block AI bots, scrapers and crawlers with a single click by ccgreg on Wednesday July 03, 2024 @06:30PM (#40870613)

I wonder why you're blocking CCBot? The Common Crawl Foundation predates LLMs and most of the 10,000 research papers using our corpus have nothing to do with Machine Learning.

Edit: Publishers Target Common Crawl In Fight Over AI Training Data https://www.wired.com/story/the-fight-against-ai-comes-to-a-...

[ Parent | on HN ]

Re:Block AI bots, scrapers and crawlers with a single click by anthonyhn on Wednesday July 03, 2024 @08:01PM (#40871214)

First off, I want to thank you and the other members of the CC Foundation, the CC data set is an incredible resource to everyone.

Much of the UA data, including CCBot, is from an upstream source[0]. I was torn on whether CCBot and other archival bots should be included in the configs, since these services are not AI bot scraping services. I've added an exclusion for CCBot[1] and the archival services from the recommended configs.

[0] https://darkvisitors.com/agents/ccbot

[1] https://github.com/anthmn/ai-bot-blocker/commit/ae0c2c40fd08...

[ Parent | on HN ]

Re:Block AI bots, scrapers and crawlers with a single click by ccgreg on Wednesday July 03, 2024 @10:47PM (#40871947)

Thank you!

[ Parent | on HN ]