AI companies cause most of traffic on forums

Re:AI companies cause most of traffic on forums by TuringNYC on Monday December 30, 2024 @01:22PM (#42552017)

>> One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-...

Are they not respecting robots.txt?

[ Parent | on HN ]

Re:AI companies cause most of traffic on forums by eesmith on Monday December 30, 2024 @01:29PM (#42552085)

Quoting the top-level link to geraspora.de:

> Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. And the best thing of all: they crawl the stupidest pages possible. Recently, both ChatGPT and Amazon were - at the same time - crawling the entire edit history of the wiki.

[ Parent | on HN ]

Re:AI companies cause most of traffic on forums by vasco on Tuesday December 31, 2024 @06:18PM (#42562600)

Edit history of a wiki sounds much more interesting than the current snapshot if you want to train a model.

[ Parent | on HN ]

Re:AI companies cause most of traffic on forums by eesmith on Wednesday January 01, 2025 @03:55AM (#42564834)

Does that information improve or worsen the training?

Does it justify the resource demands?

Who pays for those resources and who benefits?

[ Parent | on HN ]