News for hackers. Stuff that matters.

A comment on Show HN: I scraped 25M Shopify products to build a search engine. Parent comment

Re:Show HN: I scraped 25M Shopify products to build a search... by xnx on Thursday December 14, 2023 @10:58AM (#38642722)
Since ~2009 many crawlers recognize "Sitemap:" directives in robots.txt to link to sitemaps: https://en.wikipedia.org/wiki/Robots.txt#Sitemap
[ Parent | on HN ]

There are no comments on this comment yet.