News for hackers. Stuff that matters.
Re:The Geocities Torrent (~1TB of awesomeness) by corin_ on Wednesday October 27, 2010 @05:55PM (#1840627)
I imagine at least somebody will download to a server and host them all there. Might grab a new 1TB drive into one of my servers and do it if I'm bored enough...
Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @06:14PM (#1840687)
I've been doing that for the last 12 months, and the collection is significantly larger than that.

http://www.reocities.com/

Re:The Geocities Torrent (~1TB of awesomeness) by vyrotek on Wednesday October 27, 2010 @06:53PM (#1840800)
Ah ha! So here's where you've all been hiding. http://www.reocities.com/TimesSquare/1266/
Re:The Geocities Torrent (~1TB of awesomeness) by samatman on Wednesday October 27, 2010 @11:03PM (#1841513)
That page is by a young 'hacker' named Simon Liu.

I wonder if it is this Simon Liu:

http://www.computer.org/portal/web/csdl/doi/10.1109/MITP.200...

"Surviving Distributed Denial of Service Attacks"

See? This is why reocities is awesome. The Simon Lius of the world should have access to their old Descent scores.

Re:The Geocities Torrent (~1TB of awesomeness) by a_m_kelly on Wednesday October 27, 2010 @09:51PM (#1841346)
I'd love to know what sort of infrastructure you're running this on. I'm in a course for my Masters for library school that deals with similar sorts of problems in maintaining and preserving for the "long term" digital materials.

From your site it sounds like you wrote a script using wget to harvest the files and another to check them against versions that were still up. What do you do on the server end now to ensure that the files are still working correctly? Are you running periodic checksums on them or the like? Finally, are you looking for any help from an interested novice?

Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @10:21PM (#1841432)
I have a large database table that stores the md5 hashes of all the files and there is a script that can compare all of the contents of the site with the hashes in the files (and with a second copy if that's what it would come to).

Some bitrot is inevitable but I think it's under control for now.

As for help, yes, but right now I'm pretty swamped in other stuff, the next round of work on reocities will likely come after the new year.

Re:The Geocities Torrent (~1TB of awesomeness) by daeken on Wednesday October 27, 2010 @10:32PM (#1841454)
Have you considered using something like MogileFS? It'd be perfect for this sort of situation.

Let me know if you're interested in this or have any questions -- I've dealt a good bit with systems like this in the past, and would love to give you a hand.

Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @11:17PM (#1841551)
You know where to find me :)

And yes, of course I'm interested. But right now no time.

Re:The Geocities Torrent (~1TB of awesomeness) by afterburner on Thursday October 28, 2010 @01:52AM (#1841921)
It is truly amazing to see one of my old sites reappear in this form. Also, there appears to be another mirror, oocities.com
Re:The Geocities Torrent (~1TB of awesomeness) by brc on Wednesday October 27, 2010 @08:23PM (#1841098)
I checked, my very first website is thus far not there. I'll wait until you're finished, and if it doesn't appear I can send the offline backup. I've been carting it around for 10 years, laughing at the animated graphics and frames.
Re:The Geocities Torrent (~1TB of awesomeness) by arst on Wednesday October 27, 2010 @06:40PM (#1840770)
While I think the web is probably a better place with reocities than without it, slapping ads on this content feels a tad bit slimy to me. I'm curious to hear your take on the ethical and legal implications of it.
Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @06:48PM (#1840791)
The ads are actually a boost for a HN'er.

And let's suppose for a second that they were not (as they have not been in the past), reocities has cost a fairly large amount of money to date (instead of made money, as you suggest), not a single person that has asked me to remove the content has ever commented on the presence of the ads, and neither has anybody that has found their stuff again because I backed it up.

On the contrary, the reactions have been almost 100% positive with a very few exceptions.

Re:The Geocities Torrent (~1TB of awesomeness) by steve19 on Wednesday October 27, 2010 @09:32PM (#1841286)
As a former Geocities user, I am very pleased he has placed ads on the site. Hopefully the ad revenue will cover the hosting costs indefinitely.

If he makes a small profit, great! Hopefully that will encourage archivists of the future.

Re:The Geocities Torrent (~1TB of awesomeness) by arst on Wednesday October 27, 2010 @07:11PM (#1840871)
(instead of made money, as you suggest)

I apologize if that's how you read my post; I certainly didn't intend to suggest that or to belittle the amount of work and money that you've put into this project.

Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @07:15PM (#1840882)
If you put up half of the money that went in to it sight unseen then we'll call it even , deal ? ;)
Re:The Geocities Torrent (~1TB of awesomeness) by pbhjpbhj on Wednesday October 27, 2010 @08:24PM (#1841100)
This is from the Google stable of morals - if people don't know you're profiting from their copyright then it's OK.

Presumably you have no legal right to redistribute any of the Geocities stuff?

The fact you spent time and money on it is neither here nor there, that's not a moral|legal argument. Presumably there have been some exceptions (you intimate that), what sort of percentage does that amount to? If you calculate that as a true reflection of the whole population then how many people's copyright do you suppose you've infringed, knowingly, against their wishes?

If someone takes your published material and slaps ads on it and republishes without your knowledge is that all good with you? (I'm guessing you may say yes here!).

It's not just about economics, it's probably as much about moral rights. People believe that their Geocities content died and was buried.

Of course they are pretty minor "offences" (or at least appears so) across a large population - akin to being a spammer or somesuch. Actually scratch that this is simply like copying others blog posts on a massive scale.

Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @08:49PM (#1841160)
> Actually scratch that this is simply like copying others blog posts on a massive scale.

No, this is saving those blog posts from extinction.

Re:The Geocities Torrent (~1TB of awesomeness) by pbhjpbhj on Wednesday October 27, 2010 @09:01PM (#1841197)
I copied my content from Geocities to another provider and then on to my own site (eventually). I'm sure I'm not alone. Very little of that content is still live anywhere but there is a little I think.

Also, why do you get to be the arbiter of whether others content can be allowed to disappear or not? If I own the copyright then it's within my rights to have all copies destroyed, for example, you secretly (as you've not notified copyright holders AFAICT) keeping a copy is infringing my moral right to control that work.

Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @09:34PM (#1841290)
Not to toot my own horn here, but let me just give you one sample of the kind of email I get about this project, I'll leave you to judge the rest of them by this single one (it just happens to be the last one and more in this vein would be excessive and embarrassing):

--

I don't know who you are, but I just want say you are an angel. I thought I had laid the very first website I ever built on geocities to rest, and words cannot describe my utter and complete surprise to find it resurrected on reocities.

You say this project is a labor of love -- and that is exactly how I felt about my own website. It was, for at least 10 years of my life, the best expression of who I was, and it means so much to be able to relive that era again.

You are truly doing a public service by preserving early relics of Internet culture. I can only imagine that generations from now, when people are digging into the history of the web, they will be fascinated by what you've saved.

Thank you SO MUCH for doing what you do!

webmistress

reocities.com/avcorner2000

--

I'm not going to argue morality here with you any further, you apparently have a bee up your bonnet about this, but as they say, no good deed goes unpunished, there is no reason why this would be an exception.

greetings

Re:The Geocities Torrent (~1TB of awesomeness) by pbhjpbhj on Thursday October 28, 2010 @12:09PM (#1843453)
>I'm not going to argue morality here with you any further, you apparently have a bee up your bonnet about this, but as they say, no good deed goes unpunished, there is no reason why this would be an exception.

I'm arguing the application of copyright law (I'm not for the law as it stands incidentally) and for the moral rights of producers of copyright law. You appear to be arguing that it's fine to break the law because how you do it makes some people happy. The same rationale (at a different scale) makes speeding OK for teenagers if it impresses their mates and they're lucky enough not to have killed anyone yet.

The problem is that if we allow what you've done (which I don't dislike, indeed I'd consider myself an admirer in general of what I've seen of your work) for anyone then we allow copying other's blog posts adding adverts and putting them on one's own website, we allow copying books that are still in copyright and republishing them, etc..

It's a technicality but important to the case in point IMO. The email from webmistress is grateful for you saving her from not having properly backed up her work, not from having infringed copyright. If you now copy her current website and display it as your own with ads, will she be happy? I'd warrant no, not until the point at which she deletes it all by accident and comes to you because she hasn't backed up. Is this an argument against copyright, probably, not a great one but still it is one.

The fact that you're hearing the positive results is going to be largely selection bias.

If you've bothered reading then thanks for your responses and for not going ad hominem on my ass.

Peace,

pbhj

Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Thursday October 28, 2010 @01:19PM (#1843816)
The point of difference here though is that

(1) laws have been set aside many times when the net benefit for the common good outweighed the rights of the individual

(you can still disagree that that is a good thing though)

(2) such exemptions apply to libraries and other 'violators' that serve a different goal than piracy (for instance, preservation and access)

(3) in this case those that benefit the most are the original copyright holders

(4) there is a procedure in place to deal with those copyright holders that do not want their information out there

For the record, a fairly knowledgeable lawyer on copyright law in the netherlands here has reviewed the whole thing and think there is absolutely no problem defending my actions (just in case there would have been, I would have done whatever his advice would have been).

It's been up for a year, the one time someone threatened to sue (of course, some hotshot lawyer with a corporate page on geocities :) ), he backed off and became real nice once he realized that no judge was ever going to sign off on him suing for damages and whatnot without first asking politely to remove the stuff.

Laws are there to be respected. In exceptional cases - such as the going out of business of a repository of this size - you can break them if you go about it nicely and try to limit the damage as much as you can.

There are other people out there that have also made copies of all this data that have turned the whole thing in to an adsense fest complete with SEO spam tactics. That might be a better target for your anger.

Lastly, how much would you give for a copy of the library of Alexandria ?

I'm sure that geocities can not on average be compared with the quality of what was stored there but you'd be surprised by some of the stuff that I've found amongst the wreckage and we'd all be culturally poorer if it had gone to waste.

Re:The Geocities Torrent (~1TB of awesomeness) by pbhjpbhj on Thursday October 28, 2010 @02:10PM (#1844108)
>Laws are there to be respected. In exceptional cases - such as the going out of business of a repository of this size - you can break them if you go about it nicely and try to limit the damage as much as you can.

That's not how the law works here. You break the law whether you're held to account for it or not. Copyright law in Europe is stricter in many ways than in the US (WRT personal use for example).

>That might be a better target for your anger.

Grrr, I'm soooo anngggrrry. Really, I'm quite calm. /rageface

>Lastly, how much would you give for a copy of the library of Alexandria ?

A lot. Probably not my first born though. This hits at the correct route for attacking poor law. Obviously in Alexandria there was no copyright, it was all PD.

People are welcome to mark their pages PD (or some other liberal license; this is the legal procedure for your #4) and HTML5 should (does? via microformats?) allow a license (CC, PD, C, CL, FDL, whatever) to be applied and readily parsed so that you could stay within the law and still do your white knight deal-y.

Moreover those who wish for people to be able to copy without restriction should petition for a change in the law.

The law is an ass but you're stuck with it. I don't consider the value of the stuff you've saved (as much as I've seen, certainly not an in depth study) to be that high that civil disobedience should be practised in order to preserve it.

Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Thursday October 28, 2010 @07:06PM (#1845345)
> I don't consider the value of the stuff you've saved (as much as I've seen, certainly not an in depth study) to be that high that civil disobedience should be practised in order to preserve it.

And that's where we disagree. Talk to a researcher in 500 years or so to get the better reasons why compared to the ones that I can give to you today.

But what I would not give to have the nasa pages about the spaceshuttle flights that I helped put out on the net back.

Those are gone forever, I wished someone had broken copyright law to preserve them.

Re:The Geocities Torrent (~1TB of awesomeness) by pbhjpbhj on Thursday October 28, 2010 @08:02PM (#1845500)
>And that's where we disagree.

I'm sure there are plenty of examples of the type of page preserved by the likes of archive.org.

>Those are gone forever, I wished someone had broken copyright law to preserve them.

Issue all your stuff PD and then people won't need to break the law to do what you appear to want them to do.

Re:The Geocities Torrent (~1TB of awesomeness) by brianobush on Wednesday October 27, 2010 @08:32PM (#1841120)
It is digital archeology. The content was as good as gone, just someone dug it up and put a big banner ad over it and selling lemonade on the side.
Re:The Geocities Torrent (~1TB of awesomeness) by philwelch on Wednesday October 27, 2010 @09:38PM (#1841305)
The AwesomenessReminder banner ads are perfect, though. Completely fit the style and the era.
Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @09:46PM (#1841332)
Yes, I put them up there as a joke initially because Zachary came up with his animated gifs but then I thought oh, what the heck and left them up. I'm sure he's not making much money of them - if any - but it looked like a perfect fit.

His new site looks much better actually: http://www.awesomenessreminders.com/

Re:The Geocities Torrent (~1TB of awesomeness) by pbhjpbhj on Thursday October 28, 2010 @12:20PM (#1843511)
>I'm sure he's not making much money of[f] them - if any - but it looked like a perfect fit.

I think you're being disingenuous here. You've argued that you have a right to the content and to put your ads on it so why bother trying to spin that action as a minor benefit.

If it's no benefit then save yourself and everyone else the bandwidth and remove the adframe. If it is a benefit then keep it and stand by your conviction that that is justified.

PS: for your mate, the .logo {padding-left: } looks to be about 7px too much so it doesn't align with "Bring happiness [...]" strapline and the body text. I'm using FF3.6.11 on Kubuntu.

Re:The Geocities Torrent (~1TB of awesomeness) by mhd on Wednesday October 27, 2010 @07:28PM (#1840929)
And after only two random clicks, I'm at a (bad) foot fetish site. Well, it was either that, or a "Mr. T ate my <censored>"…
Re:The Geocities Torrent (~1TB of awesomeness) by bobf on Wednesday October 27, 2010 @06:22PM (#1840718)
Great idea and name!
Re:The Geocities Torrent (~1TB of awesomeness) by pavel_lishin on Wednesday October 27, 2010 @06:23PM (#1840725)
Have you grabbed the torrent they're offering?
Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @06:29PM (#1840737)
I would be very surprised if there is anything in there that is not in reocities yet but I will certainly do a comparison.

I'll have to run that against all the deletion requests as well for this content because chances are fairly large that lots of it has already been removed at the request of the owners.

Re:The Geocities Torrent (~1TB of awesomeness) by a1k0n on Wednesday October 27, 2010 @06:40PM (#1840768)
Have you tried de-duping files to save space? I imagine there's a finite number of MIDI files and "under construction" GIFs that could just be symlinked to save a ton of space...
Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @06:51PM (#1840795)
Space is not the problem, I've got plenty of that. The bigger problem is that the underlying filesystem is having some problems dealing with the total number of files.

Total disk space usage on the array that holds reocities is about 10.5 T (that includes a master copy though) and runs to 233 million files.

Re:The Geocities Torrent (~1TB of awesomeness) by bobf on Wednesday October 27, 2010 @07:21PM (#1840903)
Have you considered implementing a NoSQL solution to store the files? I've successfully done something similar with HBase.
Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @07:50PM (#1840992)
No, but that's a good idea, I'll definitely look in to that when the next round of reocities improvements rolls around.
Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Wednesday October 27, 2010 @11:04PM (#1841517)
On second reading, I don't actually see a torrent. Did I miss something?
Re:The Geocities Torrent (~1TB of awesomeness) by flipbrad on Thursday October 28, 2010 @12:49PM (#1843652)
are you worried about copyright infringement suits?
Re:The Geocities Torrent (~1TB of awesomeness) by jacquesm on Thursday October 28, 2010 @01:07PM (#1843752)
Not one bit.

It was a situation where the common good outweighed the benefits of respecting copyright law handily, besides that the only thing that changed is the machine where the content was hosted.

I do not pass it off as my own, have a very clearly established procedure for removing the data at the request of the copyright holder.

Think of it as a hosting provider going out of business and a new one taking over the corpse of the old one. That it happened without contacting the owners is simply because the vast majority of the owners does not have contact information to begin with.

See the reocities faq for some more info on this:

http://www.reocities.com/newhome/faq.html