News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by 1vuio0pswjnm7 on Monday September 13, 2021 @07:16PM (#28518145)
Does newscatcherapi provide a list of the 60-80K news sites so the customer knows what sources are actually being searched. Or even just way to determine if site X is among the 60-80K being searched.
Re:How to Crawl the Web with Scrapy by artembugara on Monday September 13, 2021 @07:26PM (#28518224)
Second one — yes. All sources, per request.

But as a general rule, we crawl all we could/know which is a news website.

Can add any niche/industry-specific websites on request.