Crawlers impact the operations of the Wikimedia projects

Posted by manuelfcreis on Wednesday April 02, 2025 @01:54AM
from the diff.wikimedia.org dept.

https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/

( 3 comments | 5 points | on HN )

Re:Crawlers impact the operations of the Wikimedia projects by anotherpaul on Wednesday April 02, 2025 @02:22AM (#43554119)

Seems like a serious issue many sites have nowadays. I really don't want this solved by Captchas everywhere.

[ Parent | on HN ]

Re:Crawlers impact the operations of the Wikimedia projects by Handy-Man on Wednesday April 02, 2025 @08:39AM (#43555971)

If it comes to it, just block all known IP blocks of cloud providers.

[ Parent | on HN ]

Re:Crawlers impact the operations of the Wikimedia projects by thcipriani on Wednesday April 02, 2025 @11:10AM (#43557568)

There are likely a number of beneficial bots[0] running from cloud providers. And residential proxies are also widely available for very determined scrapers.

[0]: <https://stuartgeiger.com/talks/2013-08-03-wikisym-levee-brea...>

[ Parent | on HN ]