It's MUCH better to lose only one drive, because we distribute the rebuild CPU task across all 20 CPUs in the tome and we can get it synced back up in a few hours. All the chattering back and forth still doesn't come anywhere close to maxing out the 10 Gbit.
For uploads, each individual pod has a 10 Gbit/sec connection, so ACTUALLY the vault has an aggregate of 200 Gbit/sec facing towards the internet. However, every byte that a pod accepts it has to ALSO retransmit that internally so that cuts the actual theoretical upload rate to 100 Gbit/sec for a properly parallelized application.
Wow. This is pretty shocking TBH. My basic ceph nodes will crush a 40gbps link doing a rebuild/rebalance. What's your bottleneck?
Bluestore and rocks DB really changes the back-end story altogether. No more POSIX filesystem junk with journals and such.
Spelling error: check for 'utlizing' -> 'utilizing'.
Dang it! We will fix, thanks.
For an example, take a look at Mellanox's cable page which lists a large number of combinations of connectors and cable types.
So in general, to call a random sfp cable/port an ethernet port would be confusing unless both parties already knew it was passing ethernet traffic.
The actual connector and cable are mostly independent of each other. The most common connector for an ethernet cable is RJ-45 (what you're referring to as "ethernet ports"). But it's just as possible to have SFP, coaxial, or USB even.
I have a question after reading the Vault overview here:
https://www.backblaze.com/blog/vault-cloud-storage-architect...
I'm curious what the bandwidth demand is when an entire host fails and you have to drop a new replacement host in, since you would have 60 Tomes in the Vault that need to be rebuilt at once. You have at least two other parity copies when a host dies (assuming no other drives in the affected host's Tomes are dead on other hosts in the Vault set), so I'm guessing you can afford to wait the handful of days it would take to rebuild the host. I'm still curious to know what rebuild speed you plan for, since I'm guessing you'd be looking at 40Gbps NICs eventually. I was surprised to see only 2x10Gbps.