News for hackers. Stuff that matters.
Re:SSD Failures in Datacenters by toyg on Thursday July 07, 2016 @06:43AM (#12048516)
You start from the common but misguided assumption that "unknowns" are an absolute evil, whereas they are just another risk factor to weigh. You offset the costs of worst-case scenario readiness with the productivity and financial gains of using the tech.

If SSDs make your business 2x as productive or 2x as remunerative, the cost of an extra backup or two is simply a blip.

> ban SSD from critical systems.

That's an over-reaction if I've ever seen one. I guess it depends what a "critical system" is for you. If it's something that has to guarantee high performance under massive load, SSDs are what you need, period. If it's something that has to guarantee 100-year data storage under a mountain, then there might be better choices, but they might not involve spinning drives either.

Re:SSD Failures in Datacenters by wtbob on Thursday July 07, 2016 @09:39AM (#12049203)
> You start from the common but misguided assumption that "unknowns" are an absolute evil, whereas they are just another risk factor to weigh.

> > ban SSD from critical systems.

> That's an over-reaction if I've ever seen one.

No, it's really not. SFJulie's point is that we don't know enough about the way SSDs fail to build a good failure model for them, and that we currently don't seem to have a good way to even detect that they are failing. If you can't build a failure model for something, you can't build a good risk model including it. That's kinda her point.

FWIW, I don't know that I fully agree with that point: it seems to me that cryptographic integrity checks and backups would suffice, but … maybe not?

> If it's something that has to guarantee high performance under massive load, SSDs are what you need, period.

We seemed to do alright four years ago, no? SSDs are not a requirement: one can get equivalent system performance, with predictable failures, by spending more money on more hard drives and CPUs to talk to them.

Re:SSD Failures in Datacenters by kabdib on Thursday July 07, 2016 @10:41AM (#12049591)
> SSDs are not a requirement: one can get equivalent system performance, with predictable failures, by spending more money on more hard drives and CPUs to talk to them.

Lots more money. Many, many more hard drives, with failure rates to match. And you're still not going to get there.

We started running a 240TB all SSD array a few months ago and we get sub-millisecond latencies on most reads and writes. When we see an excursion to 10ms that's cause for concern; 50ms and we do a root-cause. We can peg multiple 16GBit fiber channels and it's okay. And it fits in half a rack; try that with disk.

I cannot imagine doing this with spinning rust.

Re:SSD Failures in Datacenters by hga on Thursday July 07, 2016 @10:09AM (#12049391)
it seems to me that cryptographic integrity checks and backups would suffice, but … maybe not?

To me the three scariest words in engineering are "common mode failure". I personally never buy more than one hard disk at a time, so that I'll hopefully avoid something like both haven been shaken too much in their shipping container (the best explanation anyone was able to come up with for a Seagate failure pattern some time ago).

So, yes, the above, to detect and recover, but plenty of services need very up to date backups, so if I was designing such a system, I'd make sure the (near-)real time backup would be to SSDs that share nothing in common with my production ones, different manufacturer, chip sets, etc.

Or if the load could handle it ... ah, I see why Seagate is still selling real SAS enterprise disks (last time I checked, if I remember correctly, <= 1TB and up to 15K RPM). We understand those, not that they won't also fail on you.

Re:SSD Failures in Datacenters by bluecmd on Thursday July 07, 2016 @10:12AM (#12049402)
I don't get where you got the notion that you need to understand the failure model to rely on it. It's certainly good to, and it will prevent over provisioning, but a lot of storage solutions today are so complex that understanding the failure models is practically impossible. S3? Super-black box.

Basically all you need to define is the expected output. "99.9pct of my reads are within 1ms" + "The read response is correct 99.9999% of the time" and then verify. Eject the disk if that doesn't hold, restore from replica. The rate of replacements is a factor of how close you are to the true failure model.

Re:SSD Failures in Datacenters by SFJulie on Thursday July 07, 2016 @07:40AM (#12048683)
Well, what about a falsely "functioning" SSD in a system with an SLA of 99.999%. Backups may not be the answer.

Ok, it may be RAID1+0 but, what about data corruptions?

If primary fails, and secondary is "ok" what will be the state of your data? You don't roll back an image when everything is green.

The point is that in critical systems according to the SLA or the critical aspect of the mission (flight recorder, medical instruments...) silent failures, silent data corruption may not be ok.

However for 99% of the business that is making money out of SSD right now in the world (advertisement, marketing) this is not really a concern. They don't aim at an absolute reliability. Most industry just aim at good enough.

Some small islands of IT aims at doing a legally accurate book-keeping. Like banking, accounting, legal electronic archiving for instance...

This topic is covered in Signal Processing in the first chapter of the cost of False Positive and False Negative.

Security and risk management norms force to enumerate and evaluate the costs and predicting the outcomes of failures in terms of costs.

Here, the paper is saying SSD works pretty much fine all the time. But for failure, there is a lot we don't know.

How do you evaluate the cost of a risk you cannot compute?

And how does another solution that maybe slower rates in risk management with a better failure model that can be better predicted?

The scarriest part in the study is the implication that their could be propagation of failures by coupling in certain environment and that some failure could be snowballing.

SSD are affected by heat/memory usage, CPU load is affected by failures, heat rises, SSD are more likely to fail ....

They are implying that failures can propagate or spread or happen like a contamination.

1 SSD failing alone is already a problem, even small chances of coupling in failures is making the potentiality for a massive failure.

Would you take that risk with a SLA of 99.999%?

Re:SSD Failures in Datacenters by toyg on Thursday July 07, 2016 @09:11AM (#12049043)
I honestly don't believe in five-nines and all that, but my experience is admittedly limited to sectors where it's more about point-in-time backup reliability (finance management) than anything else.

As I said, it depends on what your definition of "critical system" is. For aviation, for instance, there are many other factors (shielding etc); no hardware is perfect for all uses.

> How do you evaluate the cost of a risk you cannot compute?

Easy: you come up with a worst-case scenario, make plans to address it, and the cost of implementing such plans is your cost. Say you assume you could get silent errors: you build test routines for actual data integrity and implement test systems for continuous checks. The cost of building and maintaining all that is the cost of (accepting) your risk.

Re:SSD Failures in Datacenters by hga on Thursday July 07, 2016 @10:18AM (#12049440)
Telcos, at least in times past, really tried to achieve a true 5 9s. I worked for Lucent in 2001 on an ultimately doomed product that was essential to the company's independent survival, a Media Gateway (https://en.wikipedia.org/wiki/Media_gateway) to supply the other half of their successful Softswitch Media Gateway Controller product, and my NoVa group worked with the Cincinnati people who were responsible for the 5ESS. One of my tasks was vetting the low level design of the persistent database, and it was intense at all levels. No spinning rust, that's for sure.

Point-in-time recovery is a snap compared to that domain.

Re:SSD Failures in Datacenters by marcosdumay on Thursday July 07, 2016 @12:30PM (#12050371)
Good thing you pointed at aviation.

Aviation standard goal is 9 nines of reliability, and many systems get there.