News for hackers. Stuff that matters.
Re:Robots.txt is a suicide note by tlrobinson on Tuesday May 10, 2011 @01:31AM (#2531479)
Hide the link from users?
Re:Robots.txt is a suicide note by autoreverse on Tuesday May 10, 2011 @03:00AM (#2531666)
Using a 1 pixel transparent GIF works - also worth putting in a warning for screen readers not to follow the link.

FWIW I've been using robots.txt honeypots on two sites for the past few years to deter scrapers. Approx 1,600 IP addresses have been banned. Most of these are for data centers.

Re:Robots.txt is a suicide note by derpy on Tuesday May 10, 2011 @03:10AM (#2531680)
Fuck you guy, that's pretty pricky. Is bandwidth that expensive for you?
Re:Robots.txt is a suicide note by migrantgeek on Tuesday May 10, 2011 @03:25AM (#2531715)
That's uncalled for. If he owns the data and pays for the connectivity, he should be free to do as he pleases just like I can exclude you from my home or business without justification.
Re:Robots.txt is a suicide note by autoreverse on Tuesday May 10, 2011 @07:36AM (#2532098)
I like humans browsing my sites but I certainly don't want scrapers copying my work wholesale, claiming it as their own and ranking ahead of me in SERPs.

I'm not alone :-) see http://incredibill.blogspot.com/

Re:Robots.txt is a suicide note by eli on Tuesday May 10, 2011 @07:43AM (#2532106)
This is what Project Honeypot does http://www.projecthoneypot.org/faq.php#f
Re:Robots.txt is a suicide note by pyre on Tuesday May 10, 2011 @02:08AM (#2531561)
I seem to remember there being a thread on HN about spam bot scraping for email addresses, and some of them are sophisticated enough that they can even determine if an element has been hidden through rules applied via an external CSS file.
Re:Robots.txt is a suicide note by Aloisius on Tuesday May 10, 2011 @02:55AM (#2531657)
Link a 1 pixel image and stick it in the bottle left hand corner? While someone could click on it, it is extremely unlikely.

IIRC, technically you can have a 0 pixel image in some formats (gif) though I imagine some browsers won't like that.