News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by 1vuio0pswjnm7 on Monday September 13, 2021 @07:34PM (#28518285)
Couple of nitpicks with the OP

1. Mixing the terms "scrape" and "crawl". Crawling means following URLs found in a page, automated browsing. Scraping means extracting data from a page. Pages can be "scraped" without doing any crawling. The difference between writing a "scraper" and a "spider" is significant. The later is far more complex.

2. The OP gives the reader a peek at what some Python code looks like but it does no show us what the extracted data and its preferred output format looks like. If someone reading wanted to quickly compare this solution to some other solution, e.g., "can our solution do what this one does", they would need to see what the output data and its format looks like. To test we need to know (a) the input, i.e., the example webpage, and (b) the output, i.e., the extracted data including any desired formatting. Here we are provided with (a) only.

There are no comments on this comment yet.