Re:Robots.txt is a suicide note
by GoodIntentions on Tuesday May 10, 2011 @11:14AM
(#2532737)
Wow. What arrogance.
A Good reason to honeypot if you aren't already.
It's expected. It's polite. Respect the site owner's published policy or expect to get IP banned like any other script kiddie because when a site admin see you ripping content, he isn't thinking "yay! archive team is here to do a free backup!" he thinks you are stealing his shit.
A Good reason to honeypot if you aren't already.
It's expected. It's polite. Respect the site owner's published policy or expect to get IP banned like any other script kiddie because when a site admin see you ripping content, he isn't thinking "yay! archive team is here to do a free backup!" he thinks you are stealing his shit.