News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by m_ke on Monday September 13, 2021 @09:07PM (#28518949)
Even with scrapy it's usually best to to crawl and store full html that can be scraped offline. It's too easy to miss something when scraping on the fly or drop data that might be useful in the future.

There are no comments on this comment yet.