Re:The Definition of Evil: Microsoft's Search Wars Hurt Us All
by Tichy on Monday November 23, 2009 @03:44PM
(#958089)
Is there a law against it? I thought robots.txt is more of a convention. However, Google already said that they would heed the robots.txt.
[ Parent
| on HN ]
Re:The Definition of Evil: Microsoft's Search Wars Hurt Us All
by chollida1 on Monday November 23, 2009 @04:02PM
(#958135)
> Is there a law against it?
I don't know. I'd imagine that there isn't a law specifically written for robots.txt and scraping.
If a case went to court I'd imagine it would be settled based on laws meant for things like "keep out" and "No trespassing" signs:)
Re:The Definition of Evil: Microsoft's Search Wars Hurt Us All
by itgoon on Monday November 23, 2009 @04:16PM
(#958163)
I think not heeding robots.txt would cause them more problems than it would solve.
Currently, it gets used for more than "deny/allow all" - it is selective. If I have to worry about stuff I don't want indexed being indexed, then I'm going to block all crawling. Yes, I know that wouldn't be easy, but I wouldn't be alone in my sentiment.