News for hackers. Stuff that matters.

A comment on Every BBC Essential Mix, 1993 to 2010. Parent comment

Re:Every BBC Essential Mix, 1993 to 2010 by lsb on Saturday September 04, 2010 @07:25PM (#1663646)
Using Coral CDN:

  wget -rl 0 -np -A .mp3 http://thenine.ca.nyud.net/essential/
[ Parent | on HN ]
Re:Every BBC Essential Mix, 1993 to 2010 by aidenn0 on Sunday September 05, 2010 @12:00AM (#1663968)
Nogood, checkout the robots.txt
[ Parent | on HN ]
Re:Every BBC Essential Mix, 1993 to 2010 by _delirium on Sunday September 05, 2010 @02:43AM (#1664137)
Since Coral considers itself a distributed caching proxy rather than a crawler, it doesn't respect robots.txt. It does respect the relevant cache-control headers, like "no-cache", but this site doesn't appear to set them.

However, the main problem here is that Coral doesn't cache any files >50mb, so almost none of these are cached. The few <50mb do seem to be, though, e.g.: http://thenine.ca.nyud.net/essential/1998/1998.01.01%20-%20E...

[ Parent | on HN ]
Re:Every BBC Essential Mix, 1993 to 2010 by forkqueue on Sunday September 05, 2010 @02:12AM (#1664114)
Yup, thenine.ca is blocking coral cache for .mp3 :(
[ Parent | on HN ]
Re:Every BBC Essential Mix, 1993 to 2010 by jc-denton on Sunday September 05, 2010 @04:05PM (#1665061)
-e robots=off

However aren't they copyrighted?!

[ Parent | on HN ]