News for hackers. Stuff that matters.

A comment on If we lose the Internet Archive, we're screwed – The Statesman. Parent comment

Re:If we lose the Internet Archive, we're screwed – The Stat... by toomuchtodo on Thursday April 06, 2023 @11:17AM (#35469488)
A distributed, highly resilient digital museum. The internet archive meets S3 meets BitTorrent. We’re most of the way there, just need improvements around durability, rebalancing around geographies, and the identity and metadata (item inventory) planes.

(no association with IA, just an active contributor)

[ Parent | on HN ]
Re:If we lose the Internet Archive, we're screwed – The Stat... by zachh on Thursday April 06, 2023 @11:22AM (#35469586)
Do you know if the IA is working on such projects / is there a way to get involved? Or is it mostly third parties doing so?
[ Parent | on HN ]
Re:If we lose the Internet Archive, we're screwed – The Stat... by toomuchtodo on Thursday April 06, 2023 @11:26AM (#35469652)
http://blog.archive.org/tag/distributed-web/

My personal opinion is that they’re going about this the wrong way, with IPFS, anything blockchain, etc. They can support this entire model with a DHT (supporting PKI for identity) and torrent advertising, the latest version of BitTorrent (which has great support for cross swarm file serving by hash), and RSS or other feeds for advertising the corpus or chunks of it.

I maintain a metadata copy in R2 of all IA items and their metadata, and would love to work on the ArchiveTeam equivalent of a Warrior appliance: appliance or containers starts, consumes an RSS feed to request the least seeded items, download them up to disk space allowed, and then serve downloaded item files to the swarm.

[ Parent | on HN ]
Re:If we lose the Internet Archive, we're screwed – The Stat... by zachh on Thursday April 06, 2023 @11:29AM (#35469702)
Agreed! Super helpful, thanks.
[ Parent | on HN ]
Re:If we lose the Internet Archive, we're screwed – The Stat... by boredumb on Thursday April 06, 2023 @04:25PM (#35473908)
I could imagine something like IPFS and a strong community of volunteers hosting nodes with different partitions. I'm not sure how IPFS would scale to IA sized storage, if for nothing else there would have to be a good way to segment and balance data across volunteers nodes.
[ Parent | on HN ]