News for hackers. Stuff that matters.
Re:SSD Failures in Datacenters by spudlyo on Thursday July 07, 2016 @12:20AM (#12047462)
I worked at a company that migrated to SSDs on hundreds of HP ProLiant servers. The performance was great, it really improved our DB latency. Unfortunately the RAID 1+0 setup that was carried over from the spinning rust setup didn't work for our most common failure case; corrupted writes.

The HP RAID controller failed to detect corrupted writes on what I seem to remember were Intel SSDs. The only way we learned about failed SSD drives is when we started catching large numbers of DB page checksum errors, and by that time it was too late to do anything about it.

Operationally it sucked, swapping out drives is a lot easier maintenance than having to rebuild a DB from backup.

Re:SSD Failures in Datacenters by Veratyr on Thursday July 07, 2016 @12:33AM (#12047492)
Is there a reason you were using hardware RAID rather than a software based RAID with checksumming like ZFS?

(I mean this as an honest question, I'm interested to know if there are performance or reliability gains to be had from a hardware controller)

Re:SSD Failures in Datacenters by spudlyo on Thursday July 07, 2016 @12:57AM (#12047539)
The oft repeated performance reason is that hardware RAID controllers have battery backed write cache, which really improves durable write performance; something traditional relational DBs do quite a lot of.

Also, these machines were built by DB people, who have traditionally placed a lot of value in hardware RAID controllers, and to be fair, the HP controllers worked well with HDDs. Also ZFS wasn't a realistic option for production Linux in 2013. Me personally, I'm not a fan of exotic proprietary hardware, I much prefer working with JBOD disk setups and software RAID where appropriate.

Re:SSD Failures in Datacenters by creshal on Thursday July 07, 2016 @03:56AM (#12048072)
> The oft repeated performance reason is that hardware RAID controllers have battery backed write cache, which really improves durable write performance; something traditional relational DBs do quite a lot of.

Don't all enterprise SSDs come with their own backup capacitors?

Re:SSD Failures in Datacenters by virtualwhys on Thursday July 07, 2016 @06:46AM (#12048527)
> Don't all enterprise SSDs come with their own backup capacitors?

Yes, but the cost of such drives makes rolling the dice with consumer grade drives appealing to some[1].

From the same thread, here's a word of warning from a Stackoverflow devops engineer[2] wrt to lack of capicitors on consumer grade drives and the perfect storm that can occur when UPS goes down in a power failure (unlike with spinning disks hardware RAID is unable prevent data corruption/loss in this case).

[1] https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/

[2] https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/#commen...

Re:SSD Failures in Datacenters by kpil on Thursday July 07, 2016 @09:13AM (#12049049)
I think my consumer-grade Intel S3500 SSD have some kind of power-loss protection.

I don't think that professional devices are 100% fail-proof either. I vaguely remember some test where they setup a test with forced power failures during heavy writes, and they zapped all drives but a few eventually.

Something that worried me was that the failure mode quite often seemed to be a totally bricked SSD.

Re:SSD Failures in Datacenters by creshal on Thursday July 07, 2016 @09:21AM (#12049092)
> I think my consumer-grade Intel S3500 SSD have some kind of power-loss protection.

The fine-print is important here. Crucial (IIRC) advertises some of its low-end consumer series with power-loss protection… against internal data corruption of the Flash Translation Layer. AKA "We won't completely brick the entire device if you pull power!" Err… thank you? Why are you advertising this as feature?

Re:SSD Failures in Datacenters by imtringued on Thursday July 07, 2016 @09:29AM (#12049150)
Because OCZ SSDs used to have a failure rate of more than 50% caused by powerloss.
Re:SSD Failures in Datacenters by Something1234 on Thursday July 07, 2016 @04:09PM (#12051802)
Really,if this is true I don't think I will buy another OCZ SSD. Which brand is the best for reliability right now?
Re:SSD Failures in Datacenters by creshal on Friday July 08, 2016 @03:19AM (#12054021)
OCZ already went bankrupt over that scandal and was bought up… in 2013.
Re:SSD Failures in Datacenters by baruch on Thursday July 07, 2016 @04:29AM (#12048159)
They do. You get an ack for a write before it reaches the media as long as the supercaps are deemed ok.
Re:SSD Failures in Datacenters by dmm on Thursday July 07, 2016 @08:45AM (#12048931)
Are you saying that most SSDs lose data they claimed was saved to stable storage in the event of power loss? That's crazy. Filesystems are greatly complicated by the need to be resistant to power loss.

What is the point of all that complication if the block device is just going to throw away those guarantees? If we're giving up on power loss safety I can make a super fast filesystem by making fsync a no-op and keeping everything in ram until I feel like writing it.

Re:SSD Failures in Datacenters by creshal on Thursday July 07, 2016 @09:28AM (#12049138)
> Are you saying that most SSDs lose data they claimed was saved to stable storage in the event of power loss?

Same with HDDs. That's why battery-backed hardware RAIDs (+ disabling this cache on the connected disks) are a thing in the first place.

> What is the point of all that complication if the block device is just going to throw away those guarantees?

Ask that storage manufacturers.

Re:SSD Failures in Datacenters by amelius on Thursday July 07, 2016 @06:02AM (#12048419)
So based on this, should filesystems include a "verify-writes" mount option?

Or would this functionality somehow be counteracted by a cache inside the SSD?

Perhaps filesystems should include a "ssd" mount option, because there might be more things to worry about (for example frequent writes wearing out the device).

Re:SSD Failures in Datacenters by rsync on Thursday July 07, 2016 @10:57AM (#12049704)
"So based on this, should filesystems include a "verify-writes" mount option?"

They do. You can, for instance, mount a UFS filesystem "sync", which means that all writes are synchronous. I assume extX has a similar option ...

Re:SSD Failures in Datacenters by amelius on Thursday July 07, 2016 @05:28PM (#12052195)
Afaik, "sync" means that the writes are immediate. "Verify-writes" would mean that the writes are not necessarily immedate, but that they are verified.

Also be careful with sync on SSD media; this is what the mount manpage says about the sync option:

> In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening.

Re:SSD Failures in Datacenters by eru on Thursday July 07, 2016 @08:37AM (#12048898)
The block device abstraction needs to leak more.
Re:SSD Failures in Datacenters by Keyframe on Thursday July 07, 2016 @09:09AM (#12049035)
When you have that many spinning rust drives replaced with SSDs - do you notice the change in power consumption?
Re:SSD Failures in Datacenters by imtringued on Thursday July 07, 2016 @09:35AM (#12049188)
That could go either way. Less power consumption because you don't need 10k rpm disks or more power consumption because the density still isn't there.
Re:SSD Failures in Datacenters by pcunite on Thursday July 07, 2016 @12:49AM (#12047521)
Interesting. Can you provide more details?