News for hackers. Stuff that matters.
Re:Ask HN: Why is Facebook indexed in search engine, against... by lcedp on Saturday September 14, 2013 @06:14PM (#6387274)
It's like a if/elsif/else statement.

`User-agent: *` means for all other bots that didn't match any section above.

Here are the rules for Google:

    User-agent: Googlebot
    Disallow: /ac.php
    Disallow: /ae.php
    Disallow: /ajax/
    Disallow: /album.php
    Disallow: /ap.php
    Disallow: /autologin.php
    Disallow: /checkpoint/
    Disallow: /confirmemail.php
    Disallow: /contact_importer/
    Disallow: /feeds/
    Disallow: /file_download.php
    Disallow: /l.php
    Disallow: /o.php
    Disallow: /p.php
    Disallow: /photo.php
    Disallow: /photo_comments.php
    Disallow: /photo_search.php
    Disallow: /photos.php
    Disallow: /sharer/
- meaning anything else allowed.

There are no comments on this comment yet.