Re:The State of Web Scraping 2022
by digitcatphd on Wednesday January 12, 2022 @12:49PM
(#29910290)
Google is a crawler not a scraper, these are two totally different things
[ Parent
| on HN ]
Re:The State of Web Scraping 2022
by 1vuio0pswjnm7 on Wednesday January 12, 2022 @11:28PM
(#29916873)
A crawl requires "extraction" of data from a web page, which according to Wikipedia is part of the definition of so-called "web scraping". Even if a crawler is using a sitemap.xml file, it still has to "scrape" (retrieve and extract from) that file first. It seems crawling always requires scraping.
If all the pages to be retrieved are known a priori, before retrieval begins, then one would likely call that "scraping". Whereas if not all pages are known before retrieval begins, then one would likely call that "crawling".