Re:How to Crawl the Web with Scrapy
by aynyc on Monday September 13, 2021 @04:06PM
(#28516135)
If you are just doing one or two pages, say you want to get weather for your location, then requests is sufficient. But if you want to do many pages where you might want to scan and follow, requests gets tedious very quickly.
[ Parent
| on HN ]
If you’re a web developer not really, rather than worrying about storage backendes, spiders, yielding and managing loops and items, you could just host a DRF or Flask API with your scrapers (written in Requests+lxml) initiated with an API request.
I guess it’s a matter of preference