> We're not having 119 disks read 8.5TB each and sending it all to the server with the new disk.

> Instead, 119 servers are each recovering 85GB and then sending that to the server with the new disk.

Could you explain how this is possible? In other words, how do you arrange the blocks so that each individual device has enough data to reconstruct 85GB from another drive without doing network I/O? I'm not aware of a scheme where this is possible. Can you point to a paper or describe how this scheme works?

Simple arithmetic suggests that it is not possible to rebuild 100 different 8.5GB chunks from a single 8.5TB selected from an larger amount of data encoded with Reed-Solomon (100,20). The simple arithmetic suggests that you would only be able to reconstruct a total of 1.7TB from any individual 8.5TB disk, which is only 20% of what your scheme suggests. It is also unclear to me what would happen in this scheme if two disks were lost simultaneously, which is nearly guaranteed to happen at some point if you are playing around with 120 disks.

Simple erasure coding means that e.g. with (100,20) Reed-Solomon, you spread each stripe across 120 devices, and therefore need to read from 100 different devices to recover any one lost chunk from a single stripe. There are some schemes which reduce this but the gains are modest and there are tradeoffs involving e.g. encoding time or overhead, for example, the "Hitchhiker" schemes published by researchers at Facebook:

https://www.cs.cmu.edu/~nihars/publications/Hitchhiker_SIGCO...

Let's say three drives have failed.

Each server with a working drive reads a contiguous run of 117MB off its hard drive. It then sends the first MB to server 1, the second MB to server 2, the third MB to server 3... the last MB to server 117.

Each of those servers receives 117 chunks of a single 1MB stripe.

It then does a parity calculation to figure out the missing 3 1MB chunks, and sends them to the rebuilding servers.

We have now recovered 117MB of the disk. Our network data sent was 120MB per server. Every lost chunk was recovered via sibling chunks from 100+ different devices.

You can then optimize this by having the servers only send 100 chunks of each stripe across the network.

Once you repeat this operation 85470 times, sending 100+3MB each time, you have recovered 3 10TB disks with a network use of 8.8TB per server. (The efficiency is slightly worse than the 8.5TB for recovering one disk)

So you're still sending over a petabyte through your switch, but that's what good switches are built for, handling cross-traffic from every single port at the same time. A gigabit per server is acceptable, and 10gig approaches overkill.

This seems pretty obvious to me once you say it, I guess I didn't understand what you were trying to say. It seems like that goes both ways, too... what I'm trying to say is that the amount of reconstruction work increases as the encoding size increases. You're offering suggestions for how to accomplish that work but it doesn't change the underlying fact that wider encodings require more work for reconstruction.

And that really was all I was trying to say... you don't make encodings arbitrarily wide because the space savings must be weighed against the reconstruction costs.

Not arbitrarily wide, agreed. But it's also true that you can go a lot wider than a typical raid array without much difficulty.

I guess something to keep in mind is that you need bigger chunks to read efficiently, and smaller stripes to write efficiently. But in archival storage you don't care about write speed so the balance changes a lot.