News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by aynyc on Monday September 13, 2021 @04:06PM (#28516135)
If you are just doing one or two pages, say you want to get weather for your location, then requests is sufficient. But if you want to do many pages where you might want to scan and follow, requests gets tedious very quickly.
Re:How to Crawl the Web with Scrapy by r_singh on Monday September 13, 2021 @04:33PM (#28516503)
If you’re a web developer not really, rather than worrying about storage backendes, spiders, yielding and managing loops and items, you could just host a DRF or Flask API with your scrapers (written in Requests+lxml) initiated with an API request.

I guess it’s a matter of preference