News for hackers. Stuff that matters.
Re:What Hard Drive Should I Buy? by deltaqueue on Tuesday January 21, 2014 @11:47AM (#7096401)
From the article and throughout the comments here it seems Backblaze prefers cheaper drives over a few percentage points of reliability. It would be interesting to see some data showing the tradeoff, but I suspect it reveals too much of their operation. At first glance it appears you can get a drive with .9% failure rate (HGST 7K3000) for $127[1], and yet BB really likes the WD Red, which has a higher failure rate (3.2%) and cost[2].

What might shed light without revealing too much is information about where they source drives today (their sourcing coverage during the shortage was very cool!). I suspect they're finding some nice bulk discounts somewhere.

[1] http://www.amazon.com/Hitachi-Deskstar-7K3000-HDS723030ALA64... [2] http://www.amazon.com/WD-Red-NAS-Hard-Drive/dp/B008JJLW4M/ (both seem to be market consumer prices)

Re:What Hard Drive Should I Buy? by brianwski on Tuesday January 21, 2014 @12:53PM (#7096926)
> What might shed light without revealing too much is information about where they source drives today

Backblaze employee here -> we are willing to buy from anybody, we have no loyalty. Lowest price (for a particular drive model) always wins. Once per month we ask about 20 common suppliers for their "best price". We have bought from "B&H Photo Video", NewEgg, Amazon, etc among others. We're always willing to add more possible vendors, but I think we drop you from the list if the vendor bid prices don't even come close for 3 months - that means you don't understand anything and you're wasting our time.

Re:What Hard Drive Should I Buy? by freshyill on Tuesday January 21, 2014 @01:55PM (#7097383)
With the volume you must buy in, why not buy direct from the manufacturers? I imagine they could supply you with their OEM pricing and product packaging, which seems like it would save money? Or is it a strategic reason like not getting stuck with one manufacturer?
Re:What Hard Drive Should I Buy? by grecy on Tuesday January 21, 2014 @06:10PM (#7099306)
I asked that question last time BB employees were here on HN.. the answer was they just don't buy enough volume to qualify to buy direct from the manufacturers. It sounds like you must but enormous quantities before the manufacturers will give you the time.
Re:What Hard Drive Should I Buy? by atYevP on Tuesday January 21, 2014 @11:03PM (#7100621)
Unfortunately that's still the case (Yev here, from Backblaze). Minimum orders are around 10,000 unites, and we're just not there yet. Thinking about starting a consortium though, so if anyone needs hard drives... ;-)
Re:What Hard Drive Should I Buy? by StuckInTexas on Wednesday January 22, 2014 @09:36AM (#7102691)
That is absolutely untrue. Talk to the big distributors, they will also be able to fix your incredibly unstable supply chain.
Re:What Hard Drive Should I Buy? by atYevP on Wednesday January 22, 2014 @12:48PM (#7104164)
We currently work with Distributors. I was referring to the manufacturers themselves, like buying directly from Seagate/WD. We currently work with a few different distributors to get different types of drives.
Re:What Hard Drive Should I Buy? by scholia on Tuesday January 21, 2014 @01:11PM (#7097053)
"Backblaze employee here"

CTO, unless that's changed....

Re:What Hard Drive Should I Buy? by brianwski on Tuesday January 21, 2014 @01:17PM (#7097095)
Guilty as charged. :-) CTO, head janitor, the company lived in my 1 bedroom apartment's living room for 3 years and up to the first 9 employees.
Re:What Hard Drive Should I Buy? by atYevP on Tuesday January 21, 2014 @01:15PM (#7097082)
He's so modest.
Re:What Hard Drive Should I Buy? by hga on Wednesday January 22, 2014 @01:21PM (#7104377)
BTW, I'm not sure B&H Photo Video deserves scare quotes.

If you've been doing video and/or photo stuff for a long time, you know them as a rock solid distributor: I've been buying video stuff from them since the middle '90s or so, camera stuff more recently (e.g. my first serious camera, vs. Amazon they had better selection with competitive pricing). Joel Spolsky was sufficiently impressed with this home town operation to do a fascinating write-up on them: http://www.inc.com/magazine/20090501/why-circuit-city-failed...

One other note on who to buy from: if you're just buying one or a few drives, Newegg has gotten really serious about packing. http://www.pregis.us/en-us/productsandservices/productsoluti... inflated padding inside a fitting cardboard box. I suspect this is about as good as the packaging Seagate requires to return a drive for warranty service. Don't know about B&H, but as of a couple of years ago Amazon had a horrible reputation for packing bare hard drives.

Re:What Hard Drive Should I Buy? by mzs on Tuesday January 21, 2014 @05:38PM (#7099110)
Ever run into counterfeit drives?
Re:What Hard Drive Should I Buy? by atYevP on Tuesday January 21, 2014 @11:04PM (#7100625)
Not that we're aware of no. We doubt it would pass our testing if it wasn't legit.
Re:What Hard Drive Should I Buy? by wting on Tuesday January 21, 2014 @11:57AM (#7096486)
If systems are designed with the expectation that hardware can and will fail often, better reliability drives aren't worth the cost as long as cheaper drives are relatively comparable. In addition to cost savings, your system has better robustness when it is decoupled from hardware reliability.

For example, Google's Map Reduce paper has a section on fault tolerance that goes into detail about how they handle the issue of failing workers:

http://research.google.com/archive/mapreduce.html

Re:What Hard Drive Should I Buy? by darkarmani on Tuesday January 21, 2014 @12:47PM (#7096873)
Deltaqueue just posted the prices and there is a $5.00 difference between the two. So you pay 4% more for 2.5% less annual failures. That sounds close enough to be worth paying more for less operational expense. Obviously, backblaze gets a better deal or they would be buying up the Hitachi drives instead.
Re:What Hard Drive Should I Buy? by olov on Tuesday January 21, 2014 @01:04PM (#7097000)
The AFR is 2.3 percentage points less (0.9% vs 3.2%), which in this case means that a single unit of the inferior brand is 3.5 times more likely to die during a full year of use. I'd love to see their calculations that justifies buying non-Hitachi drives.
Re:What Hard Drive Should I Buy? by cbr on Tuesday January 21, 2014 @01:51PM (#7097345)
I think percentage points is the right metric here. Spending a lot of money to cut down the frequency of a rare occurrence doesn't make sense, even if you can cut it down by 100x.
Re:What Hard Drive Should I Buy? by olov on Tuesday January 21, 2014 @02:08PM (#7097477)
"Rare" is the key here, thanks. An AFR of 3.2% is already a pretty damn long MTBF. Makes sense now!
Re:What Hard Drive Should I Buy? by karavelov on Tuesday January 21, 2014 @09:40PM (#7100361)
I have experience with hundreds of T of data stores. My opinion is very high of Hitachi 1T and 3T Deskstars. The problem is that they are not generally available - there could be months when you just could not order them.
Re:What Hard Drive Should I Buy? by at-fates-hands on Tuesday January 21, 2014 @12:20PM (#7096677)
Are you taking into consideration when a drive fails it requires work to replace it?

This could be minimal and something that in terms of budgetary considerations might be negligible - but I'm not sure.

Re:What Hard Drive Should I Buy? by brianwski on Tuesday January 21, 2014 @01:08PM (#7097031)
Backblaze employee here -> Yes, this gets a SMALL amount of allowance. The datacenter team begs us to buy the Hitachi drives even at twice the price, but it would bankrupt us. But if the Hitachis are only $2 or $3 more expensive per drive (including the failure rate in that calculation) then we're willing to buy them for the reduced hassle.

I think the calculation is replacing one drive takes about 15 minutes of work. If we have 30,000 drives and 2 percent fail, it takes 150 hours to replace those. In other words, one employee for one month of 8 hour days. Getting the failure rate down to 1 percent means you save 2 weeks of employee salary - maybe $5,000 total? The 30,000 drives costs you $4 million, so who cares about $5k here or there?

Re:What Hard Drive Should I Buy? by darkarmani on Tuesday January 21, 2014 @05:17PM (#7098950)
> I think the calculation is replacing one drive takes about 15 minutes of work.

Is that really the true cost of replacement? I would think there is also the cost of dealing with the warranty and the testing and monitoring. Here is the quote from the blog post about unsuitable drives:

> When one drive goes bad, it takes a lot of work to get the RAID back on-line if the whole RAID is made up of unreliable drives. It’s just not worth the trouble.

I don't have the time to think about this fully, but it seems similar to calculating the present value of a future cash flow, because there are other costs beyond the first replacement effort:

> Their average age shows 0.8 years, but since these are warranty replacements, we believe that they are refurbished drives that were returned by other customers and erased, so they already had some usage when we got them.

It sounds like the total cost of a failed drive is actually 1.5x, because 50% of the replacement drives also fail.

Re:What Hard Drive Should I Buy? by im3w1l on Thursday January 23, 2014 @07:50AM (#7107793)
You can probably cut the time for dealing with warranty down if you do it in bulk. They seem to have ~50 failing drives per month.
Re:What Hard Drive Should I Buy? by hiravgandhi on Tuesday January 21, 2014 @09:11PM (#7100250)
Yup. We used this paper as one of the proofs for why we could continue to rely on economy hardware.
Re:What Hard Drive Should I Buy? by thinkling on Tuesday January 21, 2014 @12:30PM (#7096739)
The price we pay as consumers isn't going to be the same as that paid by Backblaze when they buy 100 drives at a time. The fact that they pick the WD Red implies that they're getting a better deal on those, otherwise they wouldn't settle for the higher failure rates.

EDIT: comments from BB employees below actually say they often buy off the shelf from NewEgg and Amazon.

Re:What Hard Drive Should I Buy? by bluedino on Tuesday January 21, 2014 @12:40PM (#7096818)
They posted on their blog that they wouldn't get discount pricing until they hit something like a 10,000 drive order.
Re:What Hard Drive Should I Buy? by atYevP on Tuesday January 21, 2014 @01:04PM (#7097003)
It's a darn shame. We would love to pay less for hard drives. Even at our current capacity, we're still a "small fish". And since we try to run lean we likely could buy 10,000 drives and keep inventory, but that doesn't necessarily work out for us in the long run as hard drive prices tend to fall monthly.
Re:What Hard Drive Should I Buy? by atYevP on Tuesday January 21, 2014 @12:50PM (#7096901)
Yea, unfortunately we aren't large enough to work directly with the manufacturers. We do buy from distributors sometimes, but if we can get a cheaper deal online, we'll go that route every time.
Re:What Hard Drive Should I Buy? by tacticus on Tuesday January 21, 2014 @05:55PM (#7099202)
pity australian companies want to charge 60$ more for a hitachi drive over seagate.