News for hackers. Stuff that matters.
Re:American Chemical Society bans university after "spider-t... by PaulHoule on Friday April 04, 2014 @01:31PM (#7532952)
Funny, we used to do this when I was working at arXiv.org. We had incessant problems with robots that didn't obey robots.txt so we needed spider traps to keep the site from going down.

There are no comments on this comment yet.