Re:How You Can Help Save Upcoming.org, Posterous, and More
by lhl on Saturday April 20, 2013 @06:08PM
(#5583014)
As mentioned, the issue is getting around YDOD throttling (and that the shutdown is happening in 10 days).
If you have proper boxen, you don't need the VM. Here's how to get it up and running on a clean Ubuntu setup for example:
sudo apt-get install build-essential -y
sudo apt-get install git -y
sudo apt-get install libgnutls-dev -y
sudo apt-get install liblua5.1-dev -y
sudo apt-get install python-distribute -y
sudo easy_install pip
sudo pip install seesaw
git clone https://github.com/ArchiveTeam/yahoo-upcoming-grab.git
cd yahoo-upcoming-grab
./get-wget-lua.sh
run-pipeline pipeline.py [YOURNAMEHERE]
Re:How You Can Help Save Upcoming.org, Posterous, and More
by bdonlan on Saturday April 20, 2013 @10:43PM
(#5583658)
Does that download posterous as well, or only upcoming?
Re:How You Can Help Save Upcoming.org, Posterous, and More
by lhl on Sunday April 21, 2013 @01:03AM
(#5583924)
For Posterous you'd need to check out https://github.com/ArchiveTeam/posterous-grab and run that particular script I believe.
Re:How You Can Help Save Upcoming.org, Posterous, and More
by mark_olson on Saturday April 20, 2013 @05:55PM
(#5582962)
Yahoo and others either rate limit or do temporary IP bans if you access too many pages too quickly. Distributing the tools through a VM distributes the workload in a pretty easy-to-setup way.
Re:How You Can Help Save Upcoming.org, Posterous, and More
by pronoiac on Saturday April 20, 2013 @06:05PM
(#5583001)
The Internet Archive crawler is polite, going at a slow rate.
With a deadline looming so soon, a more aggressive effort is called for.
Re:How You Can Help Save Upcoming.org, Posterous, and More
by nwh on Saturday April 20, 2013 @08:01PM
(#5583293)
It does actually save quite a lot of bandwidth in the process. The giles are gzipped as they are uploaded to the Archive Teams server, so your virtual machine will do a lot more download than it does upload.
- Downloads dying web pages, and
- Uploads them to the Wayback Machine at the Internet Archive,
That means I'm not saving the Internet Archive any bandwidth at all, and this is no more efficient than them just downloading the site themselves.
The virtual machine itself is 174MB. This times however many volunteers means distributing the virtual machine is probably more stressful than the actual archiving operation.