Re:Data-Mining Wikipedia for Fun and Profit
by matkoniecz on Thursday August 19, 2021 @11:36AM
(#28235190)
Scrapping, especially on a large scale, can put a noticeable strain on servers.
Bulk downloads (database dumps) are much cheaper to serve for someone crawling millions of pages.
It gets even more significant if generation of reply is resource intensive (not sure is Wikipedia qualifying for that but complex templates may cause this).
Bulk downloads (database dumps) are much cheaper to serve for someone crawling millions of pages.
It gets even more significant if generation of reply is resource intensive (not sure is Wikipedia qualifying for that but complex templates may cause this).