Re:Wayback Machine director outlines the scale of everyone's...
by klodolph on Wednesday October 10, 2018 @03:05PM
(#18187919)
This seems pretty obvious to me once you say it, I guess I didn't understand what you were trying to say. It seems like that goes both ways, too... what I'm trying to say is that the amount of reconstruction work increases as the encoding size increases. You're offering suggestions for how to accomplish that work but it doesn't change the underlying fact that wider encodings require more work for reconstruction.
And that really was all I was trying to say... you don't make encodings arbitrarily wide because the space savings must be weighed against the reconstruction costs.
Re:Wayback Machine director outlines the scale of everyone's...
by Dylan16807 on Wednesday October 10, 2018 @04:03PM
(#18188359)
Not arbitrarily wide, agreed. But it's also true that you can go a lot wider than a typical raid array without much difficulty.
I guess something to keep in mind is that you need bigger chunks to read efficiently, and smaller stripes to write efficiently. But in archival storage you don't care about write speed so the balance changes a lot.
Each server with a working drive reads a contiguous run of 117MB off its hard drive. It then sends the first MB to server 1, the second MB to server 2, the third MB to server 3... the last MB to server 117.
Each of those servers receives 117 chunks of a single 1MB stripe.
It then does a parity calculation to figure out the missing 3 1MB chunks, and sends them to the rebuilding servers.
We have now recovered 117MB of the disk. Our network data sent was 120MB per server. Every lost chunk was recovered via sibling chunks from 100+ different devices.
You can then optimize this by having the servers only send 100 chunks of each stripe across the network.
Once you repeat this operation 85470 times, sending 100+3MB each time, you have recovered 3 10TB disks with a network use of 8.8TB per server. (The efficiency is slightly worse than the 8.5TB for recovering one disk)
So you're still sending over a petabyte through your switch, but that's what good switches are built for, handling cross-traffic from every single port at the same time. A gigabit per server is acceptable, and 10gig approaches overkill.