Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Ask HN: Does robots.txt serve a real purpose?.
Any web robots should parse your /robots.txt before accessing any site, but it's still the robot's (or its programmer's) choice to use it.
So: No, but still add it to your site and maintain it.
There are no comments on this comment yet.
Any web robots should parse your /robots.txt before accessing any site, but it's still the robot's (or its programmer's) choice to use it.
So: No, but still add it to your site and maintain it.