Re:How to Crawl the Web with Scrapy
by 1vuio0pswjnm7 on Monday September 13, 2021 @07:16PM
(#28518145)
Does newscatcherapi provide a list of the 60-80K news sites so the customer knows what sources are actually being searched. Or even just way to determine if site X is among the 60-80K being searched.
[ Parent
| on HN ]
Re:How to Crawl the Web with Scrapy
by artembugara on Monday September 13, 2021 @07:26PM
(#28518224)
Second one — yes. All sources, per request.
But as a general rule, we crawl all we could/know which is a news website.
Can add any niche/industry-specific websites on request.