News for hackers. Stuff that matters.
Re:American Chemical Society bans university after "spider-t... by specialp on Friday April 04, 2014 @10:36AM (#7531635)
I work for a (non profit) journal publisher and we do indeed cut off robot downloading but not after one click of a link. We analyze traffic to determine robot downloads. I suspect though that the whole entire university did not get cut off in this incident. Usually it is on a per IP basis and unless the University proxies all of their journal traffic through a single IP which is not common I think saying the whole university being blocked may be an exaggeration. I personally wish we had no robot monitor but then again we would get heavy spidering then of large files.
Re:American Chemical Society bans university after "spider-t... by dllthomas on Friday April 04, 2014 @11:27AM (#7531962)
Is there reason to block instead of throttling?
Re:American Chemical Society bans university after "spider-t... by specialp on Friday April 04, 2014 @02:22PM (#7533301)
We do have a CAPTCHA too before the block. Basically to get the block you have to really work hard at it. We also do not mind limited robot use for cases like downloading all papers given a search term or author but we do not want people downloading our entire corpus either. So throttling is not an option.

I think the case mentioned in the article is definitely heavy a heavy handed approach. When it comes down to it at my place we are just trying to block the wget -r's of the world.

Re:American Chemical Society bans university after "spider-t... by zAy0LfpBZLC8mAC on Friday April 04, 2014 @09:24PM (#7535587)
Is there any reason why you don't want people downloading your entire corpus?
Re:American Chemical Society bans university after "spider-t... by ceph_ on Friday April 04, 2014 @01:11PM (#7532806)
Or captcha?