Re:Every BBC Essential Mix, 1993 to 2010
by _delirium on Sunday September 05, 2010 @05:27AM
(#1664270)
As a heads up for anyone who doesn't realize how much music this is, that wget will slurp >100gb of stuff (or would, if it weren't for sever load).
Using Coral CDN:
wget -rl 0 -np -A .mp3 http://thenine.ca.nyud.net/essential/
Re:Every BBC Essential Mix, 1993 to 2010
by aidenn0 on Sunday September 05, 2010 @12:00AM
(#1663968)
Nogood, checkout the robots.txt
Re:Every BBC Essential Mix, 1993 to 2010
by _delirium on Sunday September 05, 2010 @02:43AM
(#1664137)
Since Coral considers itself a distributed caching proxy rather than a crawler, it doesn't respect robots.txt. It does respect the relevant cache-control headers, like "no-cache", but this site doesn't appear to set them.
However, the main problem here is that Coral doesn't cache any files >50mb, so almost none of these are cached. The few <50mb do seem to be, though, e.g.: http://thenine.ca.nyud.net/essential/1998/1998.01.01%20-%20E...
Re:Every BBC Essential Mix, 1993 to 2010
by forkqueue on Sunday September 05, 2010 @02:12AM
(#1664114)
Yup, thenine.ca is blocking coral cache for .mp3 :(
But this is what you probably want to do:
user@host:/path/to/EssentialMixRip09.04.10$ wget -rl 0 -np -A .mp3 http://thenine.ca/essential/