Re:AI companies cause most of traffic on forums
by denschub on Monday December 30, 2024 @01:37PM
(#42552172)
> Is it all crawlers that switch to a non-bot UA
I've observed only one of them do this with high confidence.
> how are they determining it's the same bot?
it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before.
I've observed only one of them do this with high confidence.
> how are they determining it's the same bot?
it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before.
> What non-bot UA do they claim?
Latest Chrome on Windows.