Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Show HN: I scraped 25M Shopify products to build a search engine. Parent comment
The robots.txt does not exclusively list what not to scrape.
It provides information on which parts are allowed and wich are not (disallowed).
It also provides sitemaps for crawlers as a starting point with more information (eg. which sites are available and how often are they updated, etc.)
There are no comments on this comment yet.
The robots.txt does not exclusively list what not to scrape.
It provides information on which parts are allowed and wich are not (disallowed).
It also provides sitemaps for crawlers as a starting point with more information (eg. which sites are available and how often are they updated, etc.)