Re:American Chemical Society bans university after "spider-t...
by PaulHoule on Friday April 04, 2014 @01:31PM
(#7532952)
Funny, we used to do this when I was working at arXiv.org. We had incessant problems with robots that didn't obey robots.txt so we needed spider traps to keep the site from going down.
[ Parent
| on HN ]
There are no comments on this comment yet.