Re:AI bots strain Wikimedia as bandwidth surges 50%
by recursivecaveat on Friday April 04, 2025 @02:17PM
(#43585971)
They talk about how a lot of the problem is all these bot requests hitting cold pages and especially multimedia that are not cached because they are rarely accessed. So I don't think it is likely that this activity is in direct response to user queries. I don't know why using an LLM would make people any more interested in very boring and obscure geographic and historical trivia than they would be via search engines.
Re:AI bots strain Wikimedia as bandwidth surges 50%
by ricardobeat on Friday April 04, 2025 @05:43PM
(#43587918)
> Automated bots seeking AI model training data for LLMs
Second sentence in the article.
Re:AI bots strain Wikimedia as bandwidth surges 50%
by lxgr on Saturday April 05, 2025 @10:34AM
(#43593842)
Yes, but I'm not convinced that that's actually (the majority of) what this is.
How many large models are realistically being trained on all of Wikimedia, compared to the number of people using AI search, possibly fetching sites very inefficiently (e.g. using a headless browser spun up from scratch, i.e. with a cold cache, every single time).
It's probably both cheaper and more accurate (for fast-changing content) to them to just hit Wikipedia servers every time than to special case search results pointing there and keeping a local copy.