Re:Robots.txt is a suicide note
by dalke on Tuesday May 10, 2011 @01:09AM
(#2531438)
Here's a concrete example of a site not worth archiving - back in the early days of the web (1994?) a friend of mine wrote a 4 function calculator. Each button was a hyperlink, so "34-12" meant going to the pages "?", "?3", "?34", "?34-", "?34-1", and "?34-12". That's an infinite number of web pages which should not be indexed.
[ Parent
| on HN ]
I remember some anti-spam vigilante code that told legitimate bots to go away with robots.txt, told humans what the game was about, and attacked e-mail address scrapers by presenting an infinite number of fake e-mail addresses.
"Click here for 1000 more e-mail addresses that will never bother a human!".
Re:Robots.txt is a suicide note
by rachelbythebay on Tuesday May 10, 2011 @02:05AM
(#2531553)
Not all of them were fake. Some of them were spam traps which never appeared anywhere else. Some of those spam traps were also coded so you could track down when they were served to connect the spiders to the spewers.
At least, that's the way I used to roll.
I love it!