Re:Only Google is really allowed to crawl the web
by indymike on Friday March 26, 2021 @03:44PM
(#26596117)
I used to run a fairly large job ad scraping operation. Our scraped data was used by many US state and federal job sites. "Scraping" is just using software to load a page and extracting content. "Crawling" is just load a page, find hyperlinks (hmm... a kind of content), and then crawling those links. Crawling is just a kind of scraping.
[ Parent
| on HN ]
There are no comments on this comment yet.