News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by jcun4128 on Monday September 13, 2021 @06:37PM (#28517820)
> We have to crawl about 60-80k news websites per day [0]

Can't even imagine that number... different languages or something?

Re:How to Crawl the Web with Scrapy by artembugara on Monday September 13, 2021 @06:43PM (#28517876)
Yeah, plus many of websites are quite niche news (construction news, for example)