Tell HN: My 'mystery project': I couldn't sleep, so I backed up GeoCities

Re:Tell HN: My 'mystery project': I couldn't sleep, so I bac... by soult on Tuesday October 27, 2009 @08:00AM (#905248)

Last time I checked, Geocities didn't support HTTP keep-alive. They do however support gzip compression, which wget doesn't support. Also, as you noticed, including "Googlebot" in the User-Agent works around the bw limit.

Did you archive regional versions (de.geocities.com, etc.) too? I see that (any-subdomain).reocities.com currently is just an alias for reocities.com. I have ~ 12 GB of archived data from de.geocities.com (couldn't find more links and didn't have the time later on), saved in the same format as wget (mtime set according to Last-Modified header etc.) if you are interested.

Anyway, good work!

[ Parent | on HN ]

Re:Tell HN: My 'mystery project': I couldn't sleep, so I bac... by jacquesm on Tuesday October 27, 2009 @12:06PM (#905628)

I thought about doing those too (esp. uk is large) but never got around to it, so yes, if you have it I would be much obliged.

Can you send me an email on how to receive the data ?

[ Parent | on HN ]

Re:Tell HN: My 'mystery project': I couldn't sleep, so I bac... by textfiles on Wednesday October 28, 2009 @06:34AM (#907536)

We snagged a bunch of the other domains.

[ Parent | on HN ]