Re:American Chemical Society bans university after "spider-t...
by ak217 on Friday April 04, 2014 @12:35PM
(#7532551)
Scientific publishing is not just weird, it's sinister.
That said, ACS is pretty sinister, too. They opposed PubChem (http://en.wikipedia.org/wiki/PubChem#ACS.27s_concerns) and in general don't behave like the nonprofit scientist trade organization that they present themselves as.
Re:American Chemical Society bans university after "spider-t...
by greglindahl on Friday April 04, 2014 @12:25PM
(#7532470)
In Highwire's case, they typically have a robots.txt blocking everyone but Google ... and the reason is not malice, it's inefficient software. Fetching a page once every few seconds is enough to overload their system.
Re:American Chemical Society bans university after "spider-t...
by AYBABTME on Friday April 04, 2014 @09:43PM
(#7535633)
A problem with robots is that they will scan all of a website. Normal user traffic is somewhat focused and is easy to cache. When one robot comes and scans everything at once, it brings a bunch of unpopular pages in the cache, possibly evicting more popular ones in the process. The more popular one will then need to be re-cached as requests come back for it.
If you avoid caching requests by robots, then instead you end up having to go through all the layers of your app, possibly going in the database.
In most situation, I don't think the above matters much. But I can see why it could be a worst case for some stacks.
Re:American Chemical Society bans university after "spider-t...
by batbomb on Saturday April 05, 2014 @01:12PM
(#7537641)
That's easily detectable and mitigable via a MRU cache, or no caching at all.
Re:American Chemical Society bans university after "spider-t...
by sgt101 on Saturday April 05, 2014 @08:00AM
(#7536761)
That might have been true in 1997, but in 2014 a page every few seconds is so far back in the noise that it's irrelevant. If you are getting problems from an IP it's normally that they are trying 1000's requests per second, and this isn't any halfway sensible spider, it's a dos attack. Robots.txt only stops you if you read and abide by it, and if you are getting a page every few seconds it's almost certain that the domain will not notice.
Re:American Chemical Society bans university after "spider-t...
by Blahah on Friday April 04, 2014 @11:54AM
(#7532220)
All the Atypon clients seem to have now disabled this trap (I tried the 5 biggest ones).
Re:American Chemical Society bans university after "spider-t...
by lazyjeff on Friday April 04, 2014 @11:51PM
(#7535898)
Those client lists are a bit unfair to compare: Atypon lists large publishers (Elsevier, IEEE, Oxford University Press, Taylor & Francis, ACS) while HighWire's list has a lot of individual journals (Journal of Early Childhood Research, Monthly Notices of the Royal Astronomical Society: Letters, etc.)
Atypon has [a relatively small client list](http://www.atypon.com/our-clients/featured-clients.php). Compare it to [Highwire](http://highwire.stanford.edu/lists/allsites.dtl). I'd be willing to bet that all journals hosted with Atypon share this spider trap—even journals that are supposed to be open access where spidering should be OK.
Scientific publishing is weird. Source: I work in scientific publishing.