News for hackers. Stuff that matters.
Re:Digg banned from Google? by rodion_89 on Wednesday March 20, 2013 @01:06PM (#5408590)
Their robots.txt file clearly asks to not be crawled at all.

  User-agent: *
  Disallow:
http://digg.com/robots.txt
Re:Digg banned from Google? by blauwbilgorgel on Wednesday March 20, 2013 @01:12PM (#5408644)
Incorrect. To configure your robots.txt to not be crawled at all use:

  User-agent: *
  Disallow: /
Allow indexing of everything with:

  User-agent: *
  Disallow:
It seems they are at this very moment struggling to change things.
Re:Digg banned from Google? by rodion_89 on Friday March 22, 2013 @01:03AM (#5421463)
You are totally right. Disregard everything I said.
Re:Digg banned from Google? by chenster on Wednesday March 20, 2013 @01:17PM (#5408678)
Disallow nothing -> allow everything.