News for hackers. Stuff that matters.
Re:Who exactly is crawling my site? by sergiosgc on Saturday October 19, 2013 @04:04AM (#6575732)
I don't block completely, but have for a long time configured servers to react like this:

- If it's cached content, there's no limit

- If it's non cached content and it is a bot, direct to the pool of servers assigned to uninteresting bots

- If it is an interesting bot (Google, bing, a couple others), use instead a pool of servers for interesting bots

Some bots are not only non-traffic generating, they are absurdly aggressive, consuming 99% of server resources for the hour or so needed to pull the content, regular site users be damned. A configuration like this is reasonably low maintenance while limiting the damage by bots.

There are no comments on this comment yet.