Re:Academic Torrents – Making 27TB of research data available
by zrm on Sunday August 12, 2018 @03:24PM
(#17745973)
If you're going to build something like this, especially when the data sets are typically large, this is a very strong case for erasure coding.
If you split a 5GB file into (20) 250MB chunks and then mirror each of them four times, there is a nontrivial probability that one of the chunks loses all four mirrors. Especially when you're using notoriously unreliable volunteer hosting.
If you split the same 5GB file into (20) 250MB data chunks and (60) 250MB erasure chunks, you consume the same amount of total storage but have to lose 61 chunks (>75% of all the hosts) before you lose any data.
Re:Academic Torrents – Making 27TB of research data available
by voltagex_ on Monday August 13, 2018 @03:06AM
(#17748485)
Offtopic: Have you looked at PAR2? I seem to recall it being really good but having some critical bug.
Re:Academic Torrents – Making 27TB of research data available
by mabbo on Sunday August 12, 2018 @08:21PM
(#17747244)
And here I go on a wiki spiral. Thanks!
Re:Academic Torrents – Making 27TB of research data available
by DataWraith on Sunday August 12, 2018 @12:41PM
(#17745148)
That sounds a lot like IA.BAK [1], a git-annex backed attempt at making a backup copy of the Internet Archive.
[1] https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....
Edit: https://git-annex.branchable.com/design/iabackup/ has more details.
I picture a system where a person adds themselves as a node to a certain group of files- like BitTorrent- but instead of downloading everything, they choose how much space and bandwidth they're offering and the system grabs the "best" pieces. Best in this case is about spreading the data out to improve resiliency, like any dates storage service would. It's not a human choice but an optimization problem.
If 250 people each offered 100MB of space, a 5GB file could be maintained with quadruple redundancy. Nodes come and go, and the system would minimize data transfers while aiming to maximize redundancy. Try to spread the files far and wide geographically, to put popular pieces of the data onto nodes offering more bandwidth.
Hmm. This sounds like a fun project to try...