News for hackers. Stuff that matters.
Re:What one may find in robots.txt by kjjw on Monday May 18, 2015 @06:44AM (#9563647)
Wonderful. So your users may occasionally be blocked due to you having a bit of fun. IP blocking is a terribly over broad way of stopping intruders.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @06:50AM (#9563660)
If my users want to go where I ask them not to go when they have no reason at all to go there, their problem. There are no links anywhere to those dirs, except for the robots.txt. Also, the blocks lift after a couple days. Honestly, nobody ever complained, and I'm sure it stops/hampers some attacks dead on.
Re:What one may find in robots.txt by kjjw on Monday May 18, 2015 @06:55AM (#9563675)
Users share IPs.
Re:What one may find in robots.txt by kjjw on Monday May 18, 2015 @07:06AM (#9563720)
What do you mean 'with bots'? We're talking about anyone at all hitting a link blocking the IP they have.

Some mobile networks have every person on the network originating traffic from the same IP. Some large institutions, universities, government departments, large companies have all their traffic coming from one IP.

This person has effectively created a feature that will perform a denial of service attack on their own website.

Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @07:00AM (#9563698)
The websites I manage aren't Facebook or Google sized. Or even HN sized. I don't see that as a real problem at all.
Re:What one may find in robots.txt by castell on Monday May 18, 2015 @08:24AM (#9563982)
You see it not as problem, because user don't see you (your sites). As IPv4 are getting rare, many share the same IP. So it's really a bad practice to ban IP for a longer period.
Re:What one may find in robots.txt by malka on Monday May 18, 2015 @07:01AM (#9563700)
with bots ? If an Ip does not respect the host rules, it deserves to be blocked.
Re:What one may find in robots.txt by germanier on Monday May 18, 2015 @07:58AM (#9563882)
For a start, the entire nation of Qatar shares 82.148.97.69.
Re:What one may find in robots.txt by ptaipale on Monday May 18, 2015 @08:21AM (#9563968)
And it's definitely not a good idea to run a bot from there.
Re:What one may find in robots.txt by ryanlol on Monday May 18, 2015 @11:13AM (#9564902)
That's actually not true. A bunch of people in Qatar use that IP though.
Re:What one may find in robots.txt by Nadya on Monday May 18, 2015 @12:21PM (#9565263)
Good thing a whopping 0% of my expected userbase originates from Qatar.
Re:What one may find in robots.txt by AgentME on Monday May 18, 2015 @02:12PM (#9565972)
robots.txt tells robots where not to go.
Re:What one may find in robots.txt by comeonnow on Monday May 18, 2015 @06:50AM (#9563661)
I don't understand why a user would access a randomly created directory that is only mentioned in the robots.txt file. Can you explain as to why you'd think they'd stumble across this?
Re:What one may find in robots.txt by gpvos on Monday May 18, 2015 @06:58AM (#9563690)
Curiosity?
Re:What one may find in robots.txt by protomyth on Monday May 18, 2015 @11:07AM (#9564861)
This is silly. If you have a user of your site going through your bots file and specifically going to directories listed as Disallow then you deal with that user. Blocking based on the robots.txt for a directory that doesn't exist anywhere but that file is fine. I did a two bad directory ban, it seemed to work fine.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @07:07AM (#9563725)
Curiosity killed the cat, or so they say.
Re:What one may find in robots.txt by gpvos on Monday May 18, 2015 @07:32AM (#9563805)
As long as the block lasts not more than a few days, as he says in a nearby comment, I don't think it's much of a problem.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @09:40AM (#9564321)
Oh, believe me they only last a few days at most. I don't use an infinite IP blocklist. It has like 100 IP's or so. When one goes in, one must come out. And lets just say there are enough badly behaved bots around that it doesn't take much time for 100 IP's to rotate.
Re:What one may find in robots.txt by beaumartinez on Monday May 18, 2015 @07:16AM (#9563752)
"But satisfaction brought him back."