Re:Show HN: I scraped 25M Shopify products to build a search...
by calebegg on Wednesday December 13, 2023 @07:57PM
(#38636452)
It seems sort of questionable to use the list of things to not scrape as a starting point for scraping.... I mean, I get it's not actually enforced.
[ Parent
| on HN ]
Re:Show HN: I scraped 25M Shopify products to build a search...
by das_keyboard on Thursday December 14, 2023 @05:54AM
(#38640000)
Not really sure why all the answers here are flagged, but you may be mistaken.
The robots.txt does not exclusively list what not to scrape.
It provides information on which parts are allowed and wich are not (disallowed).
It also provides sitemaps for crawlers as a starting point with more information (eg. which sites are available and how often are they updated, etc.)
Re:Show HN: I scraped 25M Shopify products to build a search...
by xnx on Thursday December 14, 2023 @10:58AM
(#38642722)
Since ~2009 many crawlers recognize "Sitemap:" directives in robots.txt to link to sitemaps: https://en.wikipedia.org/wiki/Robots.txt#Sitemap