News for hackers. Stuff that matters.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @06:27AM (#9563602)
I like to put a Disallow rule to a randomly named directory with an index.php file that blocks any IP that accesses it. Then, for a bit of added fun, I put another Disallow rule to a directory named "spamtrap" which does an .htaccess redirection to the block script on the randomly named directory.
Re:What one may find in robots.txt by asddubs on Monday May 18, 2015 @07:01AM (#9563701)
You understand that all you are doing with that is giving people possibly looking for attack vectors an attack vector, right? All an evildoer has to do is embed the blacklist directory as an image somewhere, send it to someone they want to lock out of the service, etc.
Re:What one may find in robots.txt by laumars on Monday May 18, 2015 @07:12AM (#9563736)
It wouldn't be too hard to prevent that. eg the honeypot directory name could be a hash of the originating IP. You'd need to then have a dynamic robots.txt but that's easily done.

The destination directory doesn't even need to exist. Worst case scenario you could handly the hash via your 404 handler or via .htaccess file if all of your hashes are prefixed. Those are only examples though - there's a multidude of ways you could handle the incoming request.

Re:What one may find in robots.txt by egeozcan on Monday May 18, 2015 @08:50AM (#9564097)
And then our hypothetical attacker can figure out how you generate the honeypot URL and embed an image with that URL for their visitors. Of course you can easily make it impossible to guess but you need to make sure that they can't obtain robots.txt via GET requests from a visitors browser (No Access-Control-Allow headers). Also don't forget a single visit from a network, sharing an IP would still ban all the network. And there is the rule that a GET request should not change any state. Banning an IP address is a change of state.

In short, it's just too much pain for little gain.

Maybe a login form can be served from that URL, and any attempts to login would then get the visitor banned via a session cookie / browser fingerprint combo (Easy to get around but at least then you're not blocking IP addresses).

Re:What one may find in robots.txt by laumars on Monday May 18, 2015 @10:03AM (#9564444)
> And then our hypothetical attacker can figure out how you generate the honeypot URL and embed an image with that URL for their visitors.

So add a salt. Just like you would when hashing passwords. You then make it more time consuming to crack the hash than it would be to perform a more typical denial of service attack

> you need to make sure that they can't obtain robots.txt via GET requests from a visitors browser (No Access-Control-Allow headers).

If an attacker already has control over the victims browser (to pull the robots.txt file) then they really don't need to bother with this attack.

> Also don't forget a single visit from a network, sharing an IP would still ban all the network.

Multiple users of the same site behind the same NATing really isn't that common unless you're Google / Facebook / etc. And when you're talking about those kinds of volumes then you'd have intrusion detection systems and possibly other, more sophisticated, honeypots in place to capture this kind of stuff. Also some busier sites will have to comply with PCI data security standards (and similar such as the Gambling Commision audits) which will require regular vulnerability scans (and possibly pen tests as well - depending on the strictness of the standards / audit) which will hopefully highlight weaknesses without the need to blanket ban via entrapment. And in the extremely rare instances where someone is innocently caught out, it's only a temporary ban anyway.

> Maybe a login form can be served from that URL, and any attempts to login would then get the visitor banned via a session cookie / browser fingerprint combo (Easy to get around but at least then you're not blocking IP addresses).

You can do this same method of banning with the honeypot you're arguing against!

While I don't disagree with any of your points per se, I do think you're being a little over dramatic. :)

Re:What one may find in robots.txt by egeozcan on Monday May 18, 2015 @10:30AM (#9564634)
Well, I was trying to point to the fact that how a small and not-so-useful feature can cause some unnecessary headaches. I know that it can work, just not worth it in my opinion. However, on second thought, I guess this could be implemented using less time than we used to write these comments :)

The login-form method would be a bit less silly, I thought, because it can be a POST. But, well...

Re:What one may find in robots.txt by laumars on Monday May 18, 2015 @12:12PM (#9565219)
The real benefit of a login form is you can easily seperate the casual nosy surfer from someone more interested in hacking your site.
Re:What one may find in robots.txt by cmdrfred on Monday May 18, 2015 @03:19PM (#9566316)
I'm with you on this, not for huge sites but if even 1% of sites used this behavior malicious bots would be forced to honor robots.txt
Re:What one may find in robots.txt by Yengas on Monday May 18, 2015 @11:32AM (#9565014)
Just check the Referer header.
Re:What one may find in robots.txt by laumars on Monday May 18, 2015 @12:14PM (#9565228)
You can't trust the referrer header for anything security related.
Re:What one may find in robots.txt by Yengas on Monday May 18, 2015 @02:59PM (#9566214)
Re:What one may find in robots.txt by laumars on Tuesday May 19, 2015 @04:27AM (#9568695)
Your citation does reitterate my point. I quote "However, checking the referer is considered to be a weaker from of CSRF protection."

The referrer header can be subject to all sorts of subtle edge cases such as switching between secure and unsecure content (or is it the other way around, I can't recall off hand?) which many broswers will then refuse to send a referrer header. So while checking the referrer might work most of the time, it's really not robust enough to be considered trustworthy for anything security related.

Re:What one may find in robots.txt by asddubs on Tuesday May 19, 2015 @03:52AM (#9568596)
the referer header does not get set on copied and pasted urls
Re:What one may find in robots.txt by Yengas on Tuesday May 19, 2015 @10:59AM (#9570512)
Since a csrf will come from a <img tag which is not on your site, you can check the header. So the only problem is if the user copy pastes the link from a website or clicks from some other application. Which decreases the chance of a substantial attack alot. If you are using http as laumars stated, this doesnt work. But i think this should be a pretty decent and easy to implement solution if you're using https.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @07:06AM (#9563722)
While you have a very valid point, in my use case I just ignore it. I don't see why anyone would want to block anyone else from accessing a small independent label website. Or the personal blog of a friend of mine. Or the portfolio site of another friend who is a designer. (edit)Or what real harm could come from it if it happened.
Re:What one may find in robots.txt by njharman on Monday May 18, 2015 @08:43AM (#9564069)
I agree with you (on risk is acceptable) but I also believe you underestimate the pettiness of people. I'm sure there are people who believe the ind label or your friends are enemies (for whatever real or imagined reason). But the set of those people who also have technical ability or access to technical ability to enact this "revenge" is almost certainly empty.
Re:What one may find in robots.txt by liviu on Monday May 18, 2015 @07:23AM (#9563770)
What if I put a link to this URL and google bot will follow? You will block google bot?
Re:What one may find in robots.txt by Ded7xSEoPKYNsDd on Monday May 18, 2015 @07:31AM (#9563802)
Googlebot won't follow the link because it's listed in robots.txt.
Re:What one may find in robots.txt by ceequof on Monday May 18, 2015 @10:39AM (#9564677)
Googlebot caches robots.txt for a very, very long time. If you disallow a directory it may take months for the entire googlebot fleet to start ignoring it. Google's official stance is that you should manage disallow directives through webmaster tools.
Re:What one may find in robots.txt by ryanlol on Monday May 18, 2015 @11:01AM (#9564825)
Yes, but it will index it.
Re:What one may find in robots.txt by kjjw on Monday May 18, 2015 @06:44AM (#9563647)
Wonderful. So your users may occasionally be blocked due to you having a bit of fun. IP blocking is a terribly over broad way of stopping intruders.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @06:50AM (#9563660)
If my users want to go where I ask them not to go when they have no reason at all to go there, their problem. There are no links anywhere to those dirs, except for the robots.txt. Also, the blocks lift after a couple days. Honestly, nobody ever complained, and I'm sure it stops/hampers some attacks dead on.
Re:What one may find in robots.txt by kjjw on Monday May 18, 2015 @06:55AM (#9563675)
Users share IPs.
Re:What one may find in robots.txt by kjjw on Monday May 18, 2015 @07:06AM (#9563720)
What do you mean 'with bots'? We're talking about anyone at all hitting a link blocking the IP they have.

Some mobile networks have every person on the network originating traffic from the same IP. Some large institutions, universities, government departments, large companies have all their traffic coming from one IP.

This person has effectively created a feature that will perform a denial of service attack on their own website.

Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @07:00AM (#9563698)
The websites I manage aren't Facebook or Google sized. Or even HN sized. I don't see that as a real problem at all.
Re:What one may find in robots.txt by castell on Monday May 18, 2015 @08:24AM (#9563982)
You see it not as problem, because user don't see you (your sites). As IPv4 are getting rare, many share the same IP. So it's really a bad practice to ban IP for a longer period.
Re:What one may find in robots.txt by malka on Monday May 18, 2015 @07:01AM (#9563700)
with bots ? If an Ip does not respect the host rules, it deserves to be blocked.
Re:What one may find in robots.txt by germanier on Monday May 18, 2015 @07:58AM (#9563882)
For a start, the entire nation of Qatar shares 82.148.97.69.
Re:What one may find in robots.txt by ptaipale on Monday May 18, 2015 @08:21AM (#9563968)
And it's definitely not a good idea to run a bot from there.
Re:What one may find in robots.txt by ryanlol on Monday May 18, 2015 @11:13AM (#9564902)
That's actually not true. A bunch of people in Qatar use that IP though.
Re:What one may find in robots.txt by Nadya on Monday May 18, 2015 @12:21PM (#9565263)
Good thing a whopping 0% of my expected userbase originates from Qatar.
Re:What one may find in robots.txt by AgentME on Monday May 18, 2015 @02:12PM (#9565972)
robots.txt tells robots where not to go.
Re:What one may find in robots.txt by comeonnow on Monday May 18, 2015 @06:50AM (#9563661)
I don't understand why a user would access a randomly created directory that is only mentioned in the robots.txt file. Can you explain as to why you'd think they'd stumble across this?
Re:What one may find in robots.txt by gpvos on Monday May 18, 2015 @06:58AM (#9563690)
Curiosity?
Re:What one may find in robots.txt by protomyth on Monday May 18, 2015 @11:07AM (#9564861)
This is silly. If you have a user of your site going through your bots file and specifically going to directories listed as Disallow then you deal with that user. Blocking based on the robots.txt for a directory that doesn't exist anywhere but that file is fine. I did a two bad directory ban, it seemed to work fine.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @07:07AM (#9563725)
Curiosity killed the cat, or so they say.
Re:What one may find in robots.txt by gpvos on Monday May 18, 2015 @07:32AM (#9563805)
As long as the block lasts not more than a few days, as he says in a nearby comment, I don't think it's much of a problem.
Re:What one may find in robots.txt by psykovsky on Monday May 18, 2015 @09:40AM (#9564321)
Oh, believe me they only last a few days at most. I don't use an infinite IP blocklist. It has like 100 IP's or so. When one goes in, one must come out. And lets just say there are enough badly behaved bots around that it doesn't take much time for 100 IP's to rotate.
Re:What one may find in robots.txt by beaumartinez on Monday May 18, 2015 @07:16AM (#9563752)
"But satisfaction brought him back."
Re:What one may find in robots.txt by lucb1e on Monday May 18, 2015 @07:51AM (#9563862)
Ditto. My favorite name is /porn and then see who visits it. Mostly bots, though.