News for hackers. Stuff that matters.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @07:06AM (#9563722)
While you have a very valid point, in my use case I just ignore it. I don't see why anyone would want to block anyone else from accessing a small independent label website. Or the personal blog of a friend of mine. Or the portfolio site of another friend who is a designer. (edit)Or what real harm could come from it if it happened.
Re:What one may find in robots.txt by njharman on Monday May 18, 2015 @08:43AM (#9564069)
I agree with you (on risk is acceptable) but I also believe you underestimate the pettiness of people. I'm sure there are people who believe the ind label or your friends are enemies (for whatever real or imagined reason). But the set of those people who also have technical ability or access to technical ability to enact this "revenge" is almost certainly empty.