Re:Dweb: Building a Resilient Web with WebTorrent
by GistNoesis on Tuesday August 28, 2018 @04:20PM
(#17862501)
Quite the opposite (it's like a RAID storage system), because of the error correcting scheme, you need a lot less storage to provide a good level of redundancy. You only need to store pieces of size 1/N * file size, and as long as you can get N of those pieces you can reconstruct the file. To guarantee a high level of availability, you only need that there are more than N hosts up containing distinct pieces. So typically you will use (~3 * size of file) of storage on your network, instead of (number of seeds * size of file) bits, and still provide with the same level of service as if number of seeds where seeding the whole file. So each node is able to store a lot more pieces of other files.
[ Parent
| on HN ]
Re:Dweb: Building a Resilient Web with WebTorrent
by eximius on Tuesday August 28, 2018 @09:43PM
(#17864645)
I might need to look up how they work because that's basically what my next question is: how do the chunks overlap and how does that affect the probability of a file being unrecoverable?
If I have a stupid simple scheme where I split a file into 100 pieces and replicate each piece 3 times, I have a 100C1 * p^3 chance of irrecoverability where p is the independent probability of pieces dropping out.
I guess I just want to know, given arbitrary p, what the probability of the file being gone is.
Re:Dweb: Building a Resilient Web with WebTorrent
by GistNoesis on Wednesday August 29, 2018 @02:00AM
(#17866014)
Error correction codes are quite magical.
Instead of splitting in 100 pieces each piece 3 times, you split it in 300 distinct pieces of size 1/100 * filesize, which any 100 can reconstruct the file. So basically you can drop 2/3 of the pieces and yet be able to reconstruct the file.
So as long as your nodes uptime is greater than 33% + eps, the file is not gone. The eps is quite small because of the law of great number.
For example if your node uptime is 50%, the pieces will be in average available on 300 * 50% = 150 +- C*uptimevariance /sqrt(N) nodes which is greater than 100 with probability ~ 1, so the file will be recoverable almost surely.