Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Google’s Sergey Brin: China, SOPA, Facebook Threaten the ‘Open Web’. Parent comment
robots.txt has always been based on a good faith.
All the reputable companies abide by it, but I don't doubt for a second they'd stop reading it if it started affecting their bottom line.
robots.txt has always been based on a good faith.
All the reputable companies abide by it, but I don't doubt for a second they'd stop reading it if it started affecting their bottom line.