Robots.txt is a suicide note

Re:Robots.txt is a suicide note by yuhong on Tuesday May 10, 2011 @01:46AM (#2531515)

BTW, robots.txt disables access to versions of pages already archived on the Wayback Machine. I encountered this when looking for old technotes on developer.apple.com.

[ Parent | on HN ]

Re:Robots.txt is a suicide note by slouch on Tuesday May 10, 2011 @01:39PM (#2533387)

The wayback machine is the only reason I use robots.txt:

    User-agent: ia_archiver
    Disallow: /

There is no good reason to let old (and possibly embarrassing) website designs distract from the current version of your website.

[ Parent | on HN ]

Re:Robots.txt is a suicide note by gojomo on Wednesday May 11, 2011 @03:37AM (#2535544)

Few users prefer the archived versions; only when they need information that has disappeared (or is temporarily unavailable) do they try the Wayback Machine.

Many, many web designers are delighted to be able to see their old creations – or even recover past work from the Archive when all other copies are lost to organizational upheaval and system problems.

Finally, note that by blocking archiving, you're opting out of history. Those who look back to see the web of today won't see your contributions. Alternatives, competitors, and other projects that are open to archiving will be seen and vividly remembered. Being invisible to the future may be OK for your projects, but many others prefer to part of the shared memory.

(FYI: I work on web archiving at the Internet Archive.)

[ Parent | on HN ]