News for hackers. Stuff that matters.

A comment on Robots.txt is a suicide note. Parent comment

Re:Robots.txt is a suicide note by derpy on Tuesday May 10, 2011 @03:10AM (#2531680)
Fuck you guy, that's pretty pricky. Is bandwidth that expensive for you?
[ Parent | on HN ]
Re:Robots.txt is a suicide note by migrantgeek on Tuesday May 10, 2011 @03:25AM (#2531715)
That's uncalled for. If he owns the data and pays for the connectivity, he should be free to do as he pleases just like I can exclude you from my home or business without justification.
[ Parent | on HN ]
Re:Robots.txt is a suicide note by autoreverse on Tuesday May 10, 2011 @07:36AM (#2532098)
I like humans browsing my sites but I certainly don't want scrapers copying my work wholesale, claiming it as their own and ranking ahead of me in SERPs.

I'm not alone :-) see http://incredibill.blogspot.com/

[ Parent | on HN ]