I'm curious what the bandwidth demand is when an entire host fails and you have to drop a new replacement host in, since you would have 60 Tomes in the Vault that need to be rebuilt at once. You have at least two other parity copies when a host dies (assuming no other drives in the affected host's Tomes are dead on other hosts in the Vault set), so I'm guessing you can afford to wait the handful of days it would take to rebuild the host. I'm still curious to know what rebuild speed you plan for, since I'm guessing you'd be looking at 40Gbps NICs eventually. I was surprised to see only 2x10Gbps.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by brianwski on Monday April 25, 2016 @01:11PM
(#11565658)
Brian from Backblaze here. If we do a full pod swap for a blank, we are disk / CPU limited in the rebuild, not 10 Gbit/sec limited. We have the rebuilds up to taking about 3/4ths of the 10 Gbit/sec link but we continue to try to change the software and buffering to get this higher.
It's MUCH better to lose only one drive, because we distribute the rebuild CPU task across all 20 CPUs in the tome and we can get it synced back up in a few hours. All the chattering back and forth still doesn't come anywhere close to maxing out the 10 Gbit.
For uploads, each individual pod has a 10 Gbit/sec connection, so ACTUALLY the vault has an aggregate of 200 Gbit/sec facing towards the internet. However, every byte that a pod accepts it has to ALSO retransmit that internally so that cuts the actual theoretical upload rate to 100 Gbit/sec for a properly parallelized application.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by mappu on Monday April 25, 2016 @04:52PM
(#11567259)
Backblaze use an in-house Java pseudo-RAID layer based on Reed-Solomon codes, with 17 data shards and 3 parity shards. There are some performance numbers here https://www.backblaze.com/blog/reed-solomon/ (2015).
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by nocarrier on Monday April 25, 2016 @03:05PM
(#11566486)
Thanks for the reply Brian. I'm a big fan of sharding across hosts and doing your own RS. I'm mildly surprised that you're disk and CPU limited, but there's lots of factors that can come into play with that and it sounds like you are happy with the tradeoffs which is all that really matters. I'm a big fan of what Backblaze has done. If you ever want to talk about replication strategies or similar stuff, I've built large scale storage systems in the past--please feel free to reach out to the email in my profile if you'd like to talk shop and trade notes.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by nwmcsween on Monday April 25, 2016 @09:08PM
(#11568691)
What is the random io like? If it's high why not cache with some ssds, it would help probably run with less power as well as faster and with less wear on the drives due to turning random io into sequential io.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by nocarrier on Monday April 25, 2016 @04:55PM
(#11567286)
Hmm, that's interesting. We never did 25 and went to 40 and 100. Well, 10 for normal hosts, 40 for top of rack switches and heavy i/o hosts, and 100 and beyond for future TOR and aggregate switches.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by dsp1234 on Monday April 25, 2016 @04:41PM
(#11567185)
sfp/sfp+/qsfp are just the connector types, and can terminate cables that are made of copper or fiber. This is a separate issue from what protocol (ethernet/infiniband/etc) is running over that cable.
For an example, take a look at Mellanox's cable page which lists a large number of combinations of connectors and cable types.
So in general, to call a random sfp cable/port an ethernet port would be confusing unless both parties already knew it was passing ethernet traffic.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by ecnahc515 on Tuesday April 26, 2016 @12:15AM
(#11569382)
I'm mostly mentioning this because it's something I just recently re-learned about.
The actual connector and cable are mostly independent of each other. The most common connector for an ethernet cable is RJ-45 (what you're referring to as "ethernet ports"). But it's just as possible to have SFP, coaxial, or USB even.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by daveguy on Monday April 25, 2016 @01:43PM
(#11565886)
Hey Yev or Brian,
First I think it is awesome that you guys come on HN to respond when backblaze makes the news. Thank you.
If you bought a prebuilt storage pod (from somewhere like http://www.backuppods.com/) could you start out at less than capacity and add more as you go? Say you started with one drive -- would it need to go in a specific location? When you add more would there be a significant reconfiguration process? Is it possible to hot-expand? I guess location and config has a lot to do with software involved. Has there been a blog on the software side of managing that much storage? Can it be as simple as a spanning partition to treat all drives as a single drive?
Also, are you tracking the cost of SSD drives? Do you expect cost parity soon? It seems that capacity parity is getting close, but cost parity (especially at that capacity) is farther off. Do you have an estimate of when it might be or is it "far off, check again in 5 years". I'm sure you all found the ssd endurance test at techreport interesting (http://techreport.com/review/27909/the-ssd-endurance-experim...).
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by brianwski on Monday April 25, 2016 @01:55PM
(#11565980)
Yes, and we do this internally in our QA and load testing lab. It's an expensive waste to put $200,000 worth of drives in a 20 pod vault when all you need is 1/60th of that!
> are you tracking the cost of SSD drives? Do you expect cost parity soon?
Yes we track SSDs, and this is a hotly contested topic inside Backblaze. Personally I'm predicting they "cross over" in cost effectiveness within 2 years. But I don't have any more insight than you do, and many inside Backblaze disagree with me.
When I say "cost effective" that is total cost of ownership, which includes paying for less electricity to power the SSD, possibly the higher density of SSDs, and whatever failure rates of the SSDs forcing us to purchase more (might be better or worse than hard drives). So whatever our little spread sheet kicks out as cheaper -> that's the one we will purchase!
One random nice thing about SSDs is there will be less case vibration, and the drives shouldn't be affected AT ALL by the small amount of fan vibration.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by DanielDent on Monday April 25, 2016 @03:52PM
(#11566832)
Am I correct in assuming that the performance improvements of SSDs are worthless for your use case?
It seems like the workload of Backblaze (like many "big data" workloads) is such that the extra IOPS don't actually help.
With thousands of drive spindles, and a very small portion of the data on on the spindles being needed at any given moment, I don't imagine you are actually limited by the performance of spinning rust.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by brianwski on Monday April 25, 2016 @07:39PM
(#11568270)
> the performance improvements of SSDs are worthless
No way! We are disk I/O limited right now. We think we can approximately double our performance simply by using SSDs instead of traditional 7200 RPM drives. I'm advocating for building an "SSD Pod" built with 1 or 2 TByte SSDs.
> and a very small portion of the data on on the spindles being needed at any given moment
With our original Personal Backup product that was true. Everybody was happy with waiting a few hours for their restores to complete. But with our new product line of B2 (competes with Amazon S3) then suddenly performance can matter much more, because programmers using B2 might be implementing all sorts of different access patterns. If those programmers are using B2 to implement online backup, then yes, still fine. But if the programmers are building hacker news or reddit or Dropbox the access patterns might require faster IOPS to be more responsive.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by DanielDent on Monday April 25, 2016 @09:35PM
(#11568799)
I guess the B2 product is growing quickly or is where you are looking to drive growth? Or are you using different storage pools for different products?
It's surprising to hear this. I would have thought that your backup workloads would leave enough unused IOPS that you wouldn't be anywhere close to hitting a performance ceiling.
What metrics do you optimize towards? Tail latency?
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by atYevP on Monday April 25, 2016 @01:50PM
(#11565945)
Yev here! It's an interesting question. We use the pods in a larger "Vault" configuration, so the way we use them is a bit different (basically a set of 20 pods are connected -> https://www.backblaze.com/blog/vault-cloud-storage-architect...). But, if you're buying a single pod, you could configure the software however you want, though you'd likely need the first hard drive to go on to the backplane that gets read first. We use different software so ours expects there to be 60 drives, but there's nothing stopping you from configuring Debian or whatnot to read a specific drive first.
We ARE tracking SSD costs but thus far it's not quite at the point where the Cost per GB is in our favor. We'd love to play with them once it gets close, it would neat to build an SSD pod and see how it stacks up!
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by teraflop on Monday April 25, 2016 @01:58PM
(#11566004)
> Can it be as simple as a spanning partition to treat all drives as a single drive?
Sure, but there are tradeoffs. One option is to just concatenate all the drives together, either with mdadm (grouping the drives into a RAID-0 array) or with LVM (adding them all to a single volume group). This lets you add drives very easily and efficiently, but has no redundancy. When (not if) one drive fails, your entire FS is toast.
If you care about your data, then you'll want to use RAID-6 for redundancy. But adding a drive to a RAID-6 array is very time-consuming, because all of your data has to be re-shuffled onto a new drive. And you don't want to do this without a backup -- although the reshape process is designed to be safely interruptible, it's not bug-free. (I speak from experience, having nearly lost a few TB this way not too long ago.)
You can compromise by grouping your drives into multiple RAID-6 arrays. For example, instead of using 58 data drives + 2 parity drives, you could use 6 arrays of (8 data + 2 parity). This gives you much less usable space (80% as opposed to 97%) but it allows you to add 10 drives at a time without having to re-shuffle all your data. And it means you can recover from a drive failure much more quickly.
Of course, you can also just treat the drives as separate block devices, and provide a single redundant filesystem at a higher level, using something like Ceph. I would assume Backblaze is doing something along these lines.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by zzzcpan on Monday April 25, 2016 @02:30PM
(#11566230)
> I would assume Backblaze is doing something along these lines.
They are using a regular filesystem on each drive and implement redundancy on top of regular files. It's more like Swift [1] than Ceph. Which is something parent should consider too, if he's going to deal with that much data.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by AdamJacobMuller on Monday April 25, 2016 @06:38PM
(#11567958)
Someone should if they haven't already.
But seriously I do hate namespace collisions like this.
Swift (the storage architecture) is something i've been a fan of for years, swift (the programming language) i've dabbled in a bit and its interesting for some use cases for sure (aside from iOS programming which I'm not interested in at all). The cognitive dissonance of switching back-and-forth is annoying. I do hope nobody ever writes or needs to use a swift API connector in swift, that would be painful.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by boulos on Monday April 25, 2016 @12:16PM
(#11565234)
So a hard drive weighs about 1.5 pounds, so 60 of them in a pod is about 100 pounds before you even include the pod itself. That doesn't sound particularly pleasant for the technician to install the first time...
You put 10 of those in a rack and now we're up to 1000 pounds. At what point does this become challenging for the floor?
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by atYevP on Monday April 25, 2016 @12:18PM
(#11565259)
Yev from Backblaze here -> Good question! We actually took that in to consideration when designing them. In our datacenter the floors can handle it, but it's definitely something that should be considered if you're thinking about building these out. Check with your datacenter about what their "floor spec" is :)
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by nickpsecurity on Monday April 25, 2016 @12:23PM
(#11565294)
Dropping in to say that from keeping the business financially in good shape to innovation level to opening up the hardware.... normally I'm the critic but all I can say is GREAT WORK. Good example to set and good tech. Keep it up. :)
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by atYevP on Monday April 25, 2016 @12:49PM
(#11565508)
Thanks! We really do this so that others follow suite. It's sort of happened with Facebook's Open Compute, and Netflix's Open Connect releasing some of their data. Dropbox recently brought things in-house as well, so we're hoping they start talking about their goodies too! It makes things more interesting, and inexpensive storage opens up the possibilities for a lot of cool stuff.
My favorite example is science labs. We've heard from a lot of them that they used to run experiments, collect the relevant data, and toss the rest. It was less expensive to run the experiment again than storing all of the data. That's changing now with storage being less expensive, and they're able to keep the entire experiment data set intact, and they love that because it means in the future they can take a look at all of the information surrounding their findings, and if the technology or science changes, they'll have the entire data set to play with instead of just a subset. Very cool stuff!
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by protomyth on Monday April 25, 2016 @12:27PM
(#11565337)
> That doesn't sound particularly pleasant for the technician to install the first time...
I've pretty much reconciled I'm going have to buy something from http://serverlift.com at some point if I keep doing this job and stuff keeps getting heavier.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by brianwski on Monday April 25, 2016 @12:24PM
(#11565306)
Brian from Backblaze here. Our corporate office in San Mateo is in an old wood building and we're on the second floor. We put a heavy rack full of experimental pods in a closet here, and I was worried it might sink through the floor BETWEEN the floor joists. :-) So I put a few layers of extra strong plywood under the rack to distribute the load. I'm happy to report no damage or sagging has happened (yet).
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by jonatron on Monday April 25, 2016 @12:28PM
(#11565355)
From my experience in installing a 24 drive storage server, it was too heavy for two people with all the hard drives installed. Do you use some sort of crane to lift the pods?
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by rbritton on Monday April 25, 2016 @12:54PM
(#11565547)
Thick plywood can handle a lot. My dad is a general contractor and uses sheets of 1 1/2" plywood to prevent his forklift (~24,000 lbs) from sinking into soft ground. At that weight a single sheet (per tire) crunches a little, but it holds up just fine.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by Scramblejams on Monday April 25, 2016 @01:44PM
(#11565892)
Thanks for the tip. Though Backblaze has spoiled us with so much good reliability data, I now feel completely paralyzed in the face of buying a new drive if Backblaze hasn't told me how much I can trust it!
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by eloff on Monday April 25, 2016 @12:08PM
(#11565171)
Now at $0.036/GB. So basically for the cost of storing the data for a single month in the cloud - and cloud storage is one of the more price competitive options in the cloud.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by brianwski on Monday April 25, 2016 @12:17PM
(#11565247)
I think he was referring to Amazon S3 prices which are up around the numbers he mentioned. But Amazon also has "Glacier" for 3/4 of one penny per month, and Backblaze has an S3 competitor for even less at 1/2 of one penny per month.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by pavs on Monday April 25, 2016 @04:44PM
(#11567205)
Amazon have Cloud storage, I am using it with 11TB right now. Fixed monthly price, 60 bucks a year. I have a cron script running thats sync with cloud drive twice a day. Delete, download, store as often as you want. I have seen couple of people on reddit storing 300+ tb, so you know you are at least safe till you reach that limit.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by asdz on Monday April 25, 2016 @08:36PM
(#11568536)
At the rate of 0.036/GB, I'm just paying Backblaze USD50/year. Meaning if I store more than 115.74GB per month, Backblaze is losing money? And this is the cost only for storage, it could be 80GB to break even the cost and revenue. How do you guys make sure you can sustain in the long run? (I don't want to face another 'Copy case' T.T)
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by atYevP on Tuesday April 26, 2016 @01:15PM
(#11573732)
Yev from Backblaze here -> Accurate. We live off the averages. More people store less data than store more data. A lot of folks that have multiple terabytes backed up often tell their friends with less data and eventually it all averages out. We're bootstrapped so we have to be cash-flow positive, and at our $5/month price-point we are. We've been at this for 9 years too and so far its worked out OK! No reason to charge a lot for a service if you don't have to. We might need to move stuff around or change things in the future, but so far we've never had to raise prices and instead just keep adding features.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by majke on Monday April 25, 2016 @03:45PM
(#11566790)
I've never heard about SMR before. I wonder - does it mean that a write to one area must be done in multiple disk rotations? Ie: if a sequential write to one file be aggregated into a single location on the disk or will it be spread around the whole plate. Just wondering how a software utilizing this should behave.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by codemac on Monday April 25, 2016 @04:44PM
(#11567213)
Heads up - if you've bought a high density disk recently, it's probably SMR internally anyways.
A write to one area can be done in a single disc rotation, as long as it's sequential.
SMR disks added what are called ZBC, Zoned Block Commands to the SCSI spec. The disk itself exposes the ability to interact with zones several ways:
- REPORT ZONES - What zones there are, how many, what type, and state of the zone.
- OPEN / CLOSE / FINISH ZONE - zone write state management
So essentially, it's like a drive exposes ~8 files, opened with O_APPEND, and you have to manage how you stripe your writes between those files to handle your workload. Your disk can also have zones that aren't shingled, but those seem to have been fairly unpopular.
I wonder what limitations the 6-month production run requirement put on design - it would seem you would have to be a lot more conservative in your "definition of done" (a bad decision can be expensive when you buy a half-year worht of gear).
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by ksec on Tuesday April 26, 2016 @02:38AM
(#11569868)
Whenever i see a post on Storagepod, i always fantasize how much it will cost Apple to provide free iCloud Backup to its users. If you have 600M active users and 30GB per backup @ 0.036/Gb. That is 648M, so even triple redundancy will only cost less then 2Billion.
If Apple could afford to give free OSX update, free iWork and ilife, Why not free iOS Backup?
Or at least offer a Time Capsule that i can backup my iOS devices without going to the cloud.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by scurvy on Monday April 25, 2016 @02:25PM
(#11566195)
Any reason on 60 drives instead of 70 or more? At that depth, there are several OEM designs that with that many drives (70).
No 25/40gig networking?
It's nice to point out the depth issue, as older racks are shallower than the newer versions. This blocks things like the 0U PDU channels and cable management channels. That said, the newer, deeper (and more popular) racks fit everything just fine.
How tall are your racks? How many pods are you getting in there? 3ph 50A power?
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by brianwski on Monday April 25, 2016 @08:05PM
(#11568383)
> Any reason on 60 drives instead of 70 or more?
One step at a time. :-) We think we can put a few more drives in the motherboard half of the POD without many other changes, and connect them to already existing SATA connectors on the motherboard. To get all the way to 75 drives we will need another row of port multipliers, which then means we need even more SATA cards, etc.
> No 25/40gig networking?
We're currently not able to saturate the 10 Gbit so for us it is wasted money to go faster. Remember that we run these in "vaults of 20" so we have 20 pods EACH with 10 Gbits so the vault hoovers in data at 200 Gbits/sec if you can thread the application.
> How tall are your racks?
Most of our racks are 45U tall where the PDU takes 4U, so we can fit (not necessarily power) 10 pods per rack and still fit a network switch in the 1U left at the top. In the past (with 45 drive pods) we ran 2 circuits of 30A 208V power (three phase). The two circuits are redundant - our power strips can flip over from one to the other if there is a loss of power on the currently used power. But I think they only put 8 or 9 of the 45 drive pods in a cabinet because it slightly exceeds what the datacenter likes. And now that we're moving up to all 60 drive pods, we have to rejigger the whole thing.
It is also worth mentioning that when we bring a 20 pod vault online, we put each of the 20 pods in different racks in different parts of the datacenter. This helps keep the vault completely online if any one component fails. For example, if the power strip feeding the pods in 1 rack power dies, it only takes 1 pod offline out of several different vaults, so ALL the vaults stay online both reading and writing data. (I hope that made sense.)
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by scurvy on Tuesday April 26, 2016 @12:26AM
(#11569443)
Makes sense. Thanks for the reply!
As to spreading the vaults around the DC, it sounds like that's more to prepare for ToR switch failure -- its sounds like an ATS handles your PDU and power circuit failover.
Yeah it sounds like you're mostly CPU bound calculating EC if you're going to rebalance and replace failed pods so 25 gbps ethernet probably isn't interesting (yet). That said, it's only marginally more expensive than 10gbps twinax. Interesting that you're running 10gig baseT. Interesting from a power and latency perspective, though I'd imagine you're not worried too much about latency.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by jeffmcjunkin on Monday April 25, 2016 @05:30PM
(#11567527)
exiftool says it's PowerPoint, surprisingly:
$ exiftool Build+Book+Backblaze+60+Drive.pdf | tail -n5 | head -n3
Title : 7400-0283 Rev A OMS Assy Backblaze 60 Drive
Producer : Mac OS X 10.11.3 Quartz PDFContext
Creator : PowerPoint
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by venomsnake on Monday April 25, 2016 @03:33PM
(#11566708)
Backblaze are/were bootstrapped. They don't have the money to burn to move outside of the sweetspot $/GB - so their logic starts with which size drive is expected to be cheapest per GB in the next 12 months. I remember some of their old posts explaining why they made the jump from 2 to 3tb drives - they are slightly more expensive today, but we expect them to go down in price soon.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by rtkwe on Monday April 25, 2016 @04:36PM
(#11567132)
It is but not by much. They actually provide the $/GB for a Seagate 8TB Archival SMR hdd in their SMR section. It comes to .034 $/GB vs .036 $/GB for the 4TB drives they're currently using. They've done the math it looks like and decided that the small gain doesn't make up for the time lost on overwrites.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by geerlingguy on Monday April 25, 2016 @12:01PM
(#11565110)
I accidentally misspelled 'BackBlaze' in the title; can someone fix it please?
Interesting decision by BackBlaze to stuff an extra row of drives in, lengthening the chassis to 33" instead of just under 29". It is their design, so they're free to do what they need for their own racks, but this does mean anyone trying to install a pod based on the 6.0 design couldn't use many closed-back racks or other kinds of racks.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by brianwski on Monday April 25, 2016 @12:15PM
(#11565229)
Brian from Backblaze here. Aren't most standard closed racks 36" deep? From this wikipedia article: https://en.wikipedia.org/wiki/19-inch_rack
"... A standard 19-inch server rack cabinet is typically 42u in height, 19-inch wide, and 36 inches deep ..."
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by teh_klev on Monday April 25, 2016 @12:48PM
(#11565499)
Ex-data centre bod here....from my experience, in the DC's I've worked in (in the UK and Europe), the racks used were most often 600mm(w) x 800mm(d) 42U, or 800x800 racks if the rack was densely cabled (i.e. there being a gap on either side of the 19" frame to bring in cable from the sides). These would then sit on 600x600 raised floor tiles, which was a convenient thing, geometry-wise.
But that was back in the late 90's/early 2000's, so things might have changed.
Another guideline in there that applies here is the one about not repeating the name of the company when it's already mentioned in the domain to the right of the title. That's a minor point, of course, but one that has had surprisingly good effects.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by cthulhujr on Monday April 25, 2016 @12:11PM
(#11565190)
Are they offering the 6.0 controllers, etc. with the standard length housing? I would think that's an option soon, if not now. Perhaps the 6.0 controllers have no benefit with fewer drives.
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by brianwski on Monday April 25, 2016 @12:21PM
(#11565286)
Brian from Backblaze here. Just to be absolutely clear, you cannot "buy" one of these from Backblaze. We don't sell them. You are free to take these instructions and assemble one yourself though! And so yes, you could simply drop the 6.0 controllers into an older chassis and it will work fine.
Also, the 6.0 housing will fit in most racks. If somebody can either measure their racks or include a link where the 33" chassis will no longer fit, we really want to hear about it!
Re:Storage Pod 6.0: Building a 60 Drive 480TB Storage Server
by gvb on Monday April 25, 2016 @12:24PM
(#11565311)
Per the article, the electronics are unchanged, just 3 more backplanes and one SATA card: "When looking at what’s new in Storage Pod 6.0 it would easy to say it has 60 drives and stop there. After all, the Motherboard, CPU, memory, SATA cards, and backplanes we use didn’t change from 5.0."
I have a question after reading the Vault overview here:
https://www.backblaze.com/blog/vault-cloud-storage-architect...
I'm curious what the bandwidth demand is when an entire host fails and you have to drop a new replacement host in, since you would have 60 Tomes in the Vault that need to be rebuilt at once. You have at least two other parity copies when a host dies (assuming no other drives in the affected host's Tomes are dead on other hosts in the Vault set), so I'm guessing you can afford to wait the handful of days it would take to rebuild the host. I'm still curious to know what rebuild speed you plan for, since I'm guessing you'd be looking at 40Gbps NICs eventually. I was surprised to see only 2x10Gbps.
It's MUCH better to lose only one drive, because we distribute the rebuild CPU task across all 20 CPUs in the tome and we can get it synced back up in a few hours. All the chattering back and forth still doesn't come anywhere close to maxing out the 10 Gbit.
For uploads, each individual pod has a 10 Gbit/sec connection, so ACTUALLY the vault has an aggregate of 200 Gbit/sec facing towards the internet. However, every byte that a pod accepts it has to ALSO retransmit that internally so that cuts the actual theoretical upload rate to 100 Gbit/sec for a properly parallelized application.
Wow. This is pretty shocking TBH. My basic ceph nodes will crush a 40gbps link doing a rebuild/rebalance. What's your bottleneck?
Bluestore and rocks DB really changes the back-end story altogether. No more POSIX filesystem junk with journals and such.
Spelling error: check for 'utlizing' -> 'utilizing'.
Dang it! We will fix, thanks.
For an example, take a look at Mellanox's cable page which lists a large number of combinations of connectors and cable types.
So in general, to call a random sfp cable/port an ethernet port would be confusing unless both parties already knew it was passing ethernet traffic.
http://www.mellanox.com/page/cables
The actual connector and cable are mostly independent of each other. The most common connector for an ethernet cable is RJ-45 (what you're referring to as "ethernet ports"). But it's just as possible to have SFP, coaxial, or USB even.
First I think it is awesome that you guys come on HN to respond when backblaze makes the news. Thank you.
If you bought a prebuilt storage pod (from somewhere like http://www.backuppods.com/) could you start out at less than capacity and add more as you go? Say you started with one drive -- would it need to go in a specific location? When you add more would there be a significant reconfiguration process? Is it possible to hot-expand? I guess location and config has a lot to do with software involved. Has there been a blog on the software side of managing that much storage? Can it be as simple as a spanning partition to treat all drives as a single drive?
Also, are you tracking the cost of SSD drives? Do you expect cost parity soon? It seems that capacity parity is getting close, but cost parity (especially at that capacity) is farther off. Do you have an estimate of when it might be or is it "far off, check again in 5 years". I'm sure you all found the ssd endurance test at techreport interesting (http://techreport.com/review/27909/the-ssd-endurance-experim...).
> are you tracking the cost of SSD drives? Do you expect cost parity soon?
Yes we track SSDs, and this is a hotly contested topic inside Backblaze. Personally I'm predicting they "cross over" in cost effectiveness within 2 years. But I don't have any more insight than you do, and many inside Backblaze disagree with me.
When I say "cost effective" that is total cost of ownership, which includes paying for less electricity to power the SSD, possibly the higher density of SSDs, and whatever failure rates of the SSDs forcing us to purchase more (might be better or worse than hard drives). So whatever our little spread sheet kicks out as cheaper -> that's the one we will purchase!
One random nice thing about SSDs is there will be less case vibration, and the drives shouldn't be affected AT ALL by the small amount of fan vibration.
It seems like the workload of Backblaze (like many "big data" workloads) is such that the extra IOPS don't actually help.
With thousands of drive spindles, and a very small portion of the data on on the spindles being needed at any given moment, I don't imagine you are actually limited by the performance of spinning rust.
No way! We are disk I/O limited right now. We think we can approximately double our performance simply by using SSDs instead of traditional 7200 RPM drives. I'm advocating for building an "SSD Pod" built with 1 or 2 TByte SSDs.
> and a very small portion of the data on on the spindles being needed at any given moment
With our original Personal Backup product that was true. Everybody was happy with waiting a few hours for their restores to complete. But with our new product line of B2 (competes with Amazon S3) then suddenly performance can matter much more, because programmers using B2 might be implementing all sorts of different access patterns. If those programmers are using B2 to implement online backup, then yes, still fine. But if the programmers are building hacker news or reddit or Dropbox the access patterns might require faster IOPS to be more responsive.
It's surprising to hear this. I would have thought that your backup workloads would leave enough unused IOPS that you wouldn't be anywhere close to hitting a performance ceiling.
What metrics do you optimize towards? Tail latency?
We ARE tracking SSD costs but thus far it's not quite at the point where the Cost per GB is in our favor. We'd love to play with them once it gets close, it would neat to build an SSD pod and see how it stacks up!
Sure, but there are tradeoffs. One option is to just concatenate all the drives together, either with mdadm (grouping the drives into a RAID-0 array) or with LVM (adding them all to a single volume group). This lets you add drives very easily and efficiently, but has no redundancy. When (not if) one drive fails, your entire FS is toast.
If you care about your data, then you'll want to use RAID-6 for redundancy. But adding a drive to a RAID-6 array is very time-consuming, because all of your data has to be re-shuffled onto a new drive. And you don't want to do this without a backup -- although the reshape process is designed to be safely interruptible, it's not bug-free. (I speak from experience, having nearly lost a few TB this way not too long ago.)
You can compromise by grouping your drives into multiple RAID-6 arrays. For example, instead of using 58 data drives + 2 parity drives, you could use 6 arrays of (8 data + 2 parity). This gives you much less usable space (80% as opposed to 97%) but it allows you to add 10 drives at a time without having to re-shuffle all your data. And it means you can recover from a drive failure much more quickly.
Of course, you can also just treat the drives as separate block devices, and provide a single redundant filesystem at a higher level, using something like Ceph. I would assume Backblaze is doing something along these lines.
They are using a regular filesystem on each drive and implement redundancy on top of regular files. It's more like Swift [1] than Ceph. Which is something parent should consider too, if he's going to deal with that much data.
[1] http://docs.openstack.org/developer/swift/overview_architect...
But seriously I do hate namespace collisions like this.
Swift (the storage architecture) is something i've been a fan of for years, swift (the programming language) i've dabbled in a bit and its interesting for some use cases for sure (aside from iOS programming which I'm not interested in at all). The cognitive dissonance of switching back-and-forth is annoying. I do hope nobody ever writes or needs to use a swift API connector in swift, that would be painful.
You put 10 of those in a rack and now we're up to 1000 pounds. At what point does this become challenging for the floor?
My favorite example is science labs. We've heard from a lot of them that they used to run experiments, collect the relevant data, and toss the rest. It was less expensive to run the experiment again than storing all of the data. That's changing now with storage being less expensive, and they're able to keep the entire experiment data set intact, and they love that because it means in the future they can take a look at all of the information surrounding their findings, and if the technology or science changes, they'll have the entire data set to play with instead of just a subset. Very cool stuff!
I've pretty much reconciled I'm going have to buy something from http://serverlift.com at some point if I keep doing this job and stuff keeps getting heavier.
If you build your own datacenter, you are of course responsible for the floor loading...
0: https://cloud.google.com/storage/docs/nearline
Disclosure: I work on Google Cloud (but not Nearline).
What do they have 300TB of, if it's not a rude question... ;-)
Tell your friends? :)
http://www.storagereview.com/what_is_shingled_magnetic_recor...
https://en.wikipedia.org/wiki/Shingled_magnetic_recording
A write to one area can be done in a single disc rotation, as long as it's sequential.
SMR disks added what are called ZBC, Zoned Block Commands to the SCSI spec. The disk itself exposes the ability to interact with zones several ways:
- REPORT ZONES - What zones there are, how many, what type, and state of the zone.
- OPEN / CLOSE / FINISH ZONE - zone write state management
So essentially, it's like a drive exposes ~8 files, opened with O_APPEND, and you have to manage how you stripe your writes between those files to handle your workload. Your disk can also have zones that aren't shingled, but those seem to have been fairly unpopular.
I wonder what limitations the 6-month production run requirement put on design - it would seem you would have to be a lot more conservative in your "definition of done" (a bad decision can be expensive when you buy a half-year worht of gear).
If Apple could afford to give free OSX update, free iWork and ilife, Why not free iOS Backup?
Or at least offer a Time Capsule that i can backup my iOS devices without going to the cloud.
No 25/40gig networking?
It's nice to point out the depth issue, as older racks are shallower than the newer versions. This blocks things like the 0U PDU channels and cable management channels. That said, the newer, deeper (and more popular) racks fit everything just fine.
How tall are your racks? How many pods are you getting in there? 3ph 50A power?
One step at a time. :-) We think we can put a few more drives in the motherboard half of the POD without many other changes, and connect them to already existing SATA connectors on the motherboard. To get all the way to 75 drives we will need another row of port multipliers, which then means we need even more SATA cards, etc.
> No 25/40gig networking?
We're currently not able to saturate the 10 Gbit so for us it is wasted money to go faster. Remember that we run these in "vaults of 20" so we have 20 pods EACH with 10 Gbits so the vault hoovers in data at 200 Gbits/sec if you can thread the application.
> How tall are your racks?
Most of our racks are 45U tall where the PDU takes 4U, so we can fit (not necessarily power) 10 pods per rack and still fit a network switch in the 1U left at the top. In the past (with 45 drive pods) we ran 2 circuits of 30A 208V power (three phase). The two circuits are redundant - our power strips can flip over from one to the other if there is a loss of power on the currently used power. But I think they only put 8 or 9 of the 45 drive pods in a cabinet because it slightly exceeds what the datacenter likes. And now that we're moving up to all 60 drive pods, we have to rejigger the whole thing.
It is also worth mentioning that when we bring a 20 pod vault online, we put each of the 20 pods in different racks in different parts of the datacenter. This helps keep the vault completely online if any one component fails. For example, if the power strip feeding the pods in 1 rack power dies, it only takes 1 pod offline out of several different vaults, so ALL the vaults stay online both reading and writing data. (I hope that made sense.)
As to spreading the vaults around the DC, it sounds like that's more to prepare for ToR switch failure -- its sounds like an ATS handles your PDU and power circuit failover.
Yeah it sounds like you're mostly CPU bound calculating EC if you're going to rebalance and replace failed pods so 25 gbps ethernet probably isn't interesting (yet). That said, it's only marginally more expensive than 10gbps twinax. Interesting that you're running 10gig baseT. Interesting from a power and latency perspective, though I'd imagine you're not worried too much about latency.
Backblaze is using 4TB drives, which is the current $/TB leader.
Interesting decision by BackBlaze to stuff an extra row of drives in, lengthening the chassis to 33" instead of just under 29". It is their design, so they're free to do what they need for their own racks, but this does mean anyone trying to install a pod based on the 6.0 design couldn't use many closed-back racks or other kinds of racks.
But that was back in the late 90's/early 2000's, so things might have changed.
Another guideline in there that applies here is the one about not repeating the name of the company when it's already mentioned in the domain to the right of the title. That's a minor point, of course, but one that has had surprisingly good effects.
Also, the 6.0 housing will fit in most racks. If somebody can either measure their racks or include a link where the 33" chassis will no longer fit, we really want to hear about it!
http://www.45drives.com/
Later in the bullet list of changes:
"Added 3 more backplanes and 1 more SATA card."