Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Robots.txt is a suicide note.
Things like /search?q= are in my robots.txt because crawling those pages is just a waste of everyone's time and resources.
There are no comments on this comment yet.
Things like /search?q= are in my robots.txt because crawling those pages is just a waste of everyone's time and resources.