News for hackers. Stuff that matters.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @07:00AM (#9563698)
The websites I manage aren't Facebook or Google sized. Or even HN sized. I don't see that as a real problem at all.
Re:What one may find in robots.txt by castell on Monday May 18, 2015 @08:24AM (#9563982)
You see it not as problem, because user don't see you (your sites). As IPv4 are getting rare, many share the same IP. So it's really a bad practice to ban IP for a longer period.