Re:Digg banned from Google?
by rodion_89 on Wednesday March 20, 2013 @01:06PM
(#5408590)
Their robots.txt file clearly asks to not be crawled at all.
http://digg.com/robots.txt
[ Parent
| on HN ]
Re:Digg banned from Google?
by blauwbilgorgel on Wednesday March 20, 2013 @01:12PM
(#5408644)
Incorrect. To configure your robots.txt to not be crawled at all use:
User-agent: *
Disallow: /
Allow indexing of everything with: User-agent: *
Disallow:
It seems they are at this very moment struggling to change things.You are totally right. Disregard everything I said.