News for hackers. Stuff that matters.

A comment on Chilling Effects removes itself from search engines.

Re:Chilling Effects removes itself from search engines by Animats on Saturday January 10, 2015 @05:40PM (#8868205)
Not removed from all search engines. Just Google. Bing works just fine. So do the little guys, like DuckDuckGo. "robots.txt":

    User-agent: Google-Legal-Removals
    Disallow:

    User-agent: Googlebot
    Noindex: /
    Allow: /pages

    User-agent: *
    Disallow: /
    Allow: /pages
[ Parent | on HN ]
Re:Chilling Effects removes itself from search engines by Tobu on Saturday January 10, 2015 @06:23PM (#8868341)
That /robots.txt does tell the other bots to not crawl most of the site. These results are unlikely to stay.
[ Parent | on HN ]
Re:Chilling Effects removes itself from search engines by TheDong on Sunday January 11, 2015 @02:19AM (#8869351)
The other sites probably just haven't processed the robots.txt change yet and will in the near future.
[ Parent | on HN ]