Re:SpiderKeeper: Admin UI for scrapy/open source scrapinghub
by slightwinder on Saturday March 07, 2020 @09:56AM
(#22511765)
How does this compare with a general solution like Apache Airflow?
[ Parent
| on HN ]
Re:SpiderKeeper: Admin UI for scrapy/open source scrapinghub
by troymc on Saturday March 07, 2020 @12:22PM
(#22512712)
The following is a simplification but should suffice.
Airflow is for hooking together and managing a bunch of data sources, data-processing nodes, and data sinks.
This project seems like a GUI for Scrapy, which is a web crawling framework, i.e. one kind of data source.