News for hackers. Stuff that matters.
Re:Ask PG: Are there IP blocks? by tlogan on Friday November 04, 2011 @12:57PM (#3196794)
Question: Is there are software which can be easily installed on apache or inside app to detect crawlers?

We use ipban but that is not what we want: we want a system which can easily detected "bad" crawler or "abusing" user and ban them for some time.

As of now, we have a simple script going thru apache logs and sending list us list of IP and their activity.

Re:Ask PG: Are there IP blocks? by cft on Friday November 04, 2011 @01:11PM (#3196845)
We use fail2ban (available as ubuntu package) very successfully for this. You can point it to apache log and finesse the rules by browser string, URL or whatever you want.