News for hackers. Stuff that matters.
Re:Nike.com's robots.txt by samhw on Friday February 11, 2022 @07:46AM (#30300768)
Doesn't it mostly penalise the honest bots, though? Or do some bots use robots.txt files as a proxy sitemap? I'm not that familiar with this area.
Re:Nike.com's robots.txt by cookiengineer on Friday February 11, 2022 @07:47AM (#30300776)
You can also insert a Disallow: /wp-admin/ in there and just serve the gzip bomb then :D
Re:Nike.com's robots.txt by berkes on Friday February 11, 2022 @07:54AM (#30300856)
I have anyone requesting this automatic going to fail2ban¹, but serving a bomb at that url is actually much smarter.

¹anything .php,.asp and some other tech we never use, also phpmyadmin, cgi-bin, and some more, actually.

Re:Nike.com's robots.txt by hughrr on Friday February 11, 2022 @08:05AM (#30300950)
Actually now you mention that I’m going to see if I can set a trap for our pen testers :)
Re:Nike.com's robots.txt by Shadonototra on Friday February 11, 2022 @11:57AM (#30303716)
i hope you don't ban the IPs permanently

nobody own their IP, and at some point a legitimate person will be assigned that IP

and most people who do bad things on the internet use some proxies, usually residential ones

it always makes me sad when people put such rules

Re:Nike.com's robots.txt by nixgeek on Friday February 11, 2022 @12:08PM (#30303890)
Most crawlers aren’t going to come from ASN which are populated with eyeballs.
Re:Nike.com's robots.txt by berkes on Monday February 14, 2022 @09:42AM (#30332546)
Fail2ban, by default, bans an IP for a limited time. For me that's 20mins.

This means that commonly only about 30 IPs are on the banlist. The chance that anyone using the same IP as a bot who is visiting any of the apps and sites on that server is not 0%, but tiny.

The only goal is to stop bots and worms from hammering my server. It hardly improves security; it shouldn't anyway.

Re:Nike.com's robots.txt by spookthesunset on Friday February 11, 2022 @10:59AM (#30302930)
Damn, that is a good idea. I wish I thought of that back when I ran a web forum.
Re:Nike.com's robots.txt by berkes on Monday February 14, 2022 @09:48AM (#30332608)
There are fail2ban configs for most popular software ready to use. E.g. WP fail2ban¹ is a config that bans an IP after failed login attempts. The defaults it ships with are well-balanced between security and not-locking-yourself-out.

Drupal, PHPBB, etc all have such generic config defaults online. For frameworks such as Rails, Django etc its a bit harder because the logging and url-schemes are not standard, but defined by the person developing in that framework.

¹https://wordpress.org/plugins/wp-fail2ban/

Re:Nike.com's robots.txt by alfiedotwtf on Friday February 11, 2022 @08:26AM (#30301183)
LOL that's actually really neat
Re:Nike.com's robots.txt by comonoid on Friday February 11, 2022 @11:55AM (#30303688)
Sane implementations do not read more than 500KiB of robots.txt contents.