News for hackers. Stuff that matters.
Re:Who exactly is crawling my site? by PaulHoule on Friday October 18, 2013 @07:08PM (#6574551)
It's actually a lot worse if your site is large since crawlers will make even more requests crawling a large site.

If I see that a crawler is sending me any traffic at all, I will accept that, but if the amount of traffic is zero, I put them in robots.txt and in the IP block list. Although I try hard to make sure my images are all clean creative common images, I block the robots that are there to find copyrighted images because these ones exist to give me nothing but trouble.

Re:Who exactly is crawling my site? by gabemart on Saturday October 19, 2013 @10:19AM (#6576487)
>I block the robots that are there to find copyrighted images because these ones exist to give me nothing but trouble.

I've never heard of these bots before. Where did you hear about them, and how do you block them?

Re:Who exactly is crawling my site? by leephillips on Saturday October 19, 2013 @08:48AM (#6576211)
Is there a list of good IPs to block, similar to the hosts files that people pass around?

EDIT: chrsstrm mentions something below