News for hackers. Stuff that matters.
Re:Internet Archive breached again through stolen access tokens by stavros on Sunday October 20, 2024 @11:11PM (#41900368)
I designed a system where you could say "donate this spare 2 TB of my disk space to the Internet Archive" and the IA would push 2 TB of data to you. This system also has the property that it can be reconstructed if the IA (or whatever provider) goes away.

Unfortunately, when I talked to a few archival teams (including the IA) about whether they'd be interested in using it, I either got no response or a negative one.

Re:Internet Archive breached again through stolen access tokens by 4gotunameagain on Monday October 21, 2024 @01:38AM (#41900972)
Why reinvent the wheel ?

There are so many proven distributed archiving systems, a lot of which are mentioned in these comments.

Re:Internet Archive breached again through stolen access tokens by stavros on Monday October 21, 2024 @05:44AM (#41902360)
What system of these allows me to donate a bunch of disk space to a provider of my choosing, without thinking about it afterwards?
Re:Internet Archive breached again through stolen access tokens by whywhywhywhy on Monday October 21, 2024 @05:25AM (#41902214)
Because the incentive will be archiving things they believe should be archived, you need the process to begin with what urls do you want to be archiving, then people will be incentivized for archiving the juicy stuff IA is used for and you just throw some stuff they didn't ask to archive in the remit of them storing what they want.
Re:Internet Archive breached again through stolen access tokens by stavros on Monday October 21, 2024 @08:24AM (#41903395)
I don't think that's necessarily true, I have a spare TB that I'd be glad to donate to the IA to store whatever they want in.
Re:Internet Archive breached again through stolen access tokens by boramalper on Monday October 21, 2024 @05:57AM (#41902442)
Is this open source or do you have any design docs? I love the idea and would love to learn more about it.
Re:Internet Archive breached again through stolen access tokens by stavros on Monday October 21, 2024 @06:00AM (#41902462)
The idea is that it'll be open source, I have a rough design doc here:

https://docs.google.com/document/d/1qKgIjUTef-I-BLWjn4sEIbYo...

I'll write up a more detailed article on it, though, it'll be good to at least have the doc public somewhere.

Re:Internet Archive breached again through stolen access tokens by TZubiri on Monday October 21, 2024 @10:11AM (#41904379)
Hosting something at a volunteer's drive, without any guarantees, is pretty useless. They can cease hosting, or have disk damage, and you lose data
Re:Internet Archive breached again through stolen access tokens by stavros on Monday October 21, 2024 @10:48AM (#41904787)
You're essentially saying that having an extra copy of data is equally as reliable as not having an extra copy of data. I would encourage you to think about this a bit more.
Re:Internet Archive breached again through stolen access tokens by TZubiri on Monday October 21, 2024 @11:30AM (#41905250)
For these parameters, yes.

If you have a raid, then you have 2 copies with like 99.99% availability and 5 mean time years to failure.

With a volunteer drive you have like ?% availability and ?% years to failure? You can't depend on it.

Also the average value of data is very low, you don't want to be making many copies of for no reason.

Re:Internet Archive breached again through stolen access tokens by stavros on Monday October 21, 2024 @11:41AM (#41905351)
That would mean that even with a million volunteer drives storing a file, you still wouldn't be able to depend on them, which is plainly wrong.

> Also the average value of data is very low, you don't want to be making many copies of for no reason.

The reason is that the value of that data is high to the archivist, since they want to preserve it.

Re:Internet Archive breached again through stolen access tokens by TZubiri on Monday October 21, 2024 @01:34PM (#41906362)
A million is out of the parameters of the case.

Realistically you won't get enough volunteer-storage to cover one IA. And even if you did, it wouldn't satisfy the mission requirements, which is to store reliably for decades all of the data.

Re:Internet Archive breached again through stolen access tokens by stavros on Monday October 21, 2024 @01:39PM (#41906413)
This isn't meant to be storage for IA, it's meant to be a distributed backup.
Re:Internet Archive breached again through stolen access tokens by TZubiri on Monday October 21, 2024 @03:28PM (#41907552)
Ah my bad, so it's not a replacement of IA. In that case it makes sense
Re:Internet Archive breached again through stolen access tokens by stavros on Monday October 21, 2024 @05:00PM (#41908483)
Yes, the idea is that this is a replacement for the torrents they make public. In case the IA goes away, we'll have this distributed dataset to fall back on.
Re:Internet Archive breached again through stolen access tokens by TZubiri on Monday October 21, 2024 @06:26PM (#41909221)
An archive of an archive