Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Ask HN: Why is Facebook indexed in search engine, against robots.txt rules?.
`User-agent: *` means for all other bots that didn't match any section above.
Here are the rules for Google:
User-agent: Googlebot Disallow: /ac.php Disallow: /ae.php Disallow: /ajax/ Disallow: /album.php Disallow: /ap.php Disallow: /autologin.php Disallow: /checkpoint/ Disallow: /confirmemail.php Disallow: /contact_importer/ Disallow: /feeds/ Disallow: /file_download.php Disallow: /l.php Disallow: /o.php Disallow: /p.php Disallow: /photo.php Disallow: /photo_comments.php Disallow: /photo_search.php Disallow: /photos.php Disallow: /sharer/
There are no comments on this comment yet.
`User-agent: *` means for all other bots that didn't match any section above.
Here are the rules for Google:
- meaning anything else allowed.