News for hackers. Stuff that matters.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by nocarrier on Monday April 25, 2016 @12:37PM (#11565424)
It's always great to see storage hardware being opened up, thanks Backblaze.

I have a question after reading the Vault overview here:

https://www.backblaze.com/blog/vault-cloud-storage-architect...

I'm curious what the bandwidth demand is when an entire host fails and you have to drop a new replacement host in, since you would have 60 Tomes in the Vault that need to be rebuilt at once. You have at least two other parity copies when a host dies (assuming no other drives in the affected host's Tomes are dead on other hosts in the Vault set), so I'm guessing you can afford to wait the handful of days it would take to rebuild the host. I'm still curious to know what rebuild speed you plan for, since I'm guessing you'd be looking at 40Gbps NICs eventually. I was surprised to see only 2x10Gbps.

Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by brianwski on Monday April 25, 2016 @01:11PM (#11565658)
Brian from Backblaze here. If we do a full pod swap for a blank, we are disk / CPU limited in the rebuild, not 10 Gbit/sec limited. We have the rebuilds up to taking about 3/4ths of the 10 Gbit/sec link but we continue to try to change the software and buffering to get this higher.

It's MUCH better to lose only one drive, because we distribute the rebuild CPU task across all 20 CPUs in the tome and we can get it synced back up in a few hours. All the chattering back and forth still doesn't come anywhere close to maxing out the 10 Gbit.

For uploads, each individual pod has a 10 Gbit/sec connection, so ACTUALLY the vault has an aggregate of 200 Gbit/sec facing towards the internet. However, every byte that a pod accepts it has to ALSO retransmit that internally so that cuts the actual theoretical upload rate to 100 Gbit/sec for a properly parallelized application.

Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by scurvy on Monday April 25, 2016 @03:34PM (#11566718)
> If we do a full pod swap for a blank, we are disk / CPU limited in the rebuild, not 10 Gbit/sec limited

Wow. This is pretty shocking TBH. My basic ceph nodes will crush a 40gbps link doing a rebuild/rebalance. What's your bottleneck?

Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by mappu on Monday April 25, 2016 @04:52PM (#11567259)
Backblaze use an in-house Java pseudo-RAID layer based on Reed-Solomon codes, with 17 data shards and 3 parity shards. There are some performance numbers here https://www.backblaze.com/blog/reed-solomon/ (2015).
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by nwmcsween on Monday April 25, 2016 @09:05PM (#11568680)
I wouldn't mind understanding why, I'm assuming at the time ceph wasn't mature enough (and still somewhat is due to btrfs, yes I know it's optional).
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by scurvy on Tuesday April 26, 2016 @12:29AM (#11569448)
btrfs isn't a requirement nor really a recommendation for production ceph clusters. It was really only there for weirdo benchers.

Bluestore and rocks DB really changes the back-end story altogether. No more POSIX filesystem junk with journals and such.

Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by nocarrier on Monday April 25, 2016 @03:05PM (#11566486)
Thanks for the reply Brian. I'm a big fan of sharding across hosts and doing your own RS. I'm mildly surprised that you're disk and CPU limited, but there's lots of factors that can come into play with that and it sounds like you are happy with the tradeoffs which is all that really matters. I'm a big fan of what Backblaze has done. If you ever want to talk about replication strategies or similar stuff, I've built large scale storage systems in the past--please feel free to reach out to the email in my profile if you'd like to talk shop and trade notes.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by nwmcsween on Monday April 25, 2016 @09:08PM (#11568691)
What is the random io like? If it's high why not cache with some ssds, it would help probably run with less power as well as faster and with less wear on the drives due to turning random io into sequential io.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by vessenes on Monday April 25, 2016 @02:38PM (#11566288)
Thanks for the blogging and open designs, it's inspiring!

Spelling error: check for 'utlizing' -> 'utilizing'.

Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by brianwski on Monday April 25, 2016 @02:50PM (#11566365)
> 'utlizing' -> 'utilizing'

Dang it! We will fix, thanks.

Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by scurvy on Monday April 25, 2016 @03:32PM (#11566702)
FYI, 25gbps ethernet is gaining in popularity much quicker than 40gbps ethernet. Mostly from cloud/scale-out stuff.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by nocarrier on Monday April 25, 2016 @04:55PM (#11567286)
Hmm, that's interesting. We never did 25 and went to 40 and 100. Well, 10 for normal hosts, 40 for top of rack switches and heavy i/o hosts, and 100 and beyond for future TOR and aggregate switches.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by scurvy on Monday April 25, 2016 @06:41PM (#11567975)
It's mainly cost savings due to the simplicity of 25gbps ethernet vs 40gbps. This article has a good run-down: http://searchnetworking.techtarget.com/feature/25-Gigabit-Et...
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by pavs on Monday April 25, 2016 @04:23PM (#11567042)
I always get confused with these, but are sfp/sfp+ ports also called ethernet ports? For whatever reason I associate ethernet with cat5/6 cable ports.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by dsp1234 on Monday April 25, 2016 @04:41PM (#11567185)
sfp/sfp+/qsfp are just the connector types, and can terminate cables that are made of copper or fiber. This is a separate issue from what protocol (ethernet/infiniband/etc) is running over that cable.

For an example, take a look at Mellanox's cable page which lists a large number of combinations of connectors and cable types.

So in general, to call a random sfp cable/port an ethernet port would be confusing unless both parties already knew it was passing ethernet traffic.

http://www.mellanox.com/page/cables

Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by pavs on Monday April 25, 2016 @04:45PM (#11567217)
Thanks for the explanation.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server by ecnahc515 on Tuesday April 26, 2016 @12:15AM (#11569382)
I'm mostly mentioning this because it's something I just recently re-learned about.

The actual connector and cable are mostly independent of each other. The most common connector for an ethernet cable is RJ-45 (what you're referring to as "ethernet ports"). But it's just as possible to have SFP, coaxial, or USB even.