News for hackers. Stuff that matters.
Re:Why I'm usually unnerved when modern SSDs die on us by Waterluvian on Tuesday December 11, 2018 @10:39AM (#18655977)
"When a HD died early, you could also imagine undetected manufacturing flaws that finally gave way. With SSDs, at least in theory that shouldn't happen"

Why shouldn't it? Isn't it just hardware too?

"With spinning HDs, drives might die abruptly but you could at least construct narratives about what could have happened to do that"

Why can't you do the same with SSDs?

It feels like the author's main complaint is the frustration of not understanding SSD hardware as well.

Is this a valid complaint? Are SSDs magical in some way? I'm not an expert but... It's just hardware with pieces that do stuff. Why can't we come up with an understanding of why it fails?

Re:Why I'm usually unnerved when modern SSDs die on us by rsync on Tuesday December 11, 2018 @11:29AM (#18656412)
"It feels like the author's main complaint is the frustration of not understanding SSD hardware as well."

What is so frustrating about SSDs is how very poorly they compare to previous incarnations of solid state storage.

Using Disk-On-Chip and/or IDE-pin-compatible CF cards, I had many, many devices in the field that lasted, mounted read-only, for decades An entire sect of the computing industry came to rely on these parts as alternatives to spinning media that could not mechanically fail.

This is not the case with SSDs at all. They fail left and right, even mounted read-only, for all manner of complicated and interesting reasons. It's very frustrating that SSDs are not a step forward in reliability from spinning media and are a step downward compared to (for instance) a 16MB consumer CF card from Sandisk, circa 2000.

rsync.net filers, which need a boot mirror, are always constructed with two unrelated SSDs - usually one Intel part and one Samsung part - so that when the inevitable usage-related failure occurs, it does not occur simultaneously to both members of the mirror which have, being a mirror, been subjected to identical usage-lives.[1]

We shouldn't have to do that.

[1] I can't overstate this - if you need a RAID mirror, do not use identical SSDs for the two members of the mirror. There are many, many cases of SSDs failing not due to "wear" or end-of-life, but due to weird usage edge cases that cause them to puke ... and in a mirror, you give the two parts identical usage ... we either get two different generations of Intel part (current gen and just-previous gen) or we get current Intel and current Samsung ...

Re:Why I'm usually unnerved when modern SSDs die on us by endorphone on Tuesday December 11, 2018 @04:45PM (#18659217)
Every failure of an SSD feels like an exceptional event. Some harbinger of doom that needs to be shouted from the rooftops. The prions of storage.

But magnetic hard drives failed all the time. I have a giant stack in my office closet just from my dev machines over the years. But it wasn't new and scary -- it was just a hard drive failing -- so it was just normal. Some had controllers fail, suddenly blinking out of existence. Another had cache memory corrupt so it just gave ridiculous readings occasionally. Others had physical failures.

I don't know where to begin relative to prior flash memory (e.g. CF cards) which were absolutely notorious trash.

It is worth noting that every smartphone the world over has an "SSD" in it. We spend remarkably little of our mental power concerned about the flash storage. It is the cause of a negligible amount of device failures.

Re:Why I'm usually unnerved when modern SSDs die on us by userbinator on Tuesday December 11, 2018 @08:47PM (#18660701)
compared to (for instance) a 16MB consumer CF card from Sandisk, circa 2000.

That would almost certainly be small-block SLC flash rated for 100K program/erase cycles and 10 years of retention. The huge-block TLC now is ~1K program/erase cycles and 2-3 years of of retention depending on where you look (the manufacturers are not surprisingly quite reluctant to release these specs...)

Re:Why I'm usually unnerved when modern SSDs die on us by Mister_Snuggles on Tuesday December 11, 2018 @11:48AM (#18656587)
I've heard very similar advice for non-SSD mirrors too. Use different manufacturers or, at the very least, use different batches of disks from the same manufacturer.
Re:Why I'm usually unnerved when modern SSDs die on us by blattimwind on Tuesday December 11, 2018 @02:32PM (#18658099)
Most importantly, never assemble an array from drives that took the same shipping path i.e. came out of the same parcel.
Re:Why I'm usually unnerved when modern SSDs die on us by lsc on Wednesday December 12, 2018 @04:51AM (#18662602)
Most people use different batches but the same manufacturer, mostly because they remember advice from back when you could get a hardware raid card that would synchronize your scsi drives.

Also, though, because most serious RAIDs contain more drives than you can find manufacturers of hard drives.

Re:Why I'm usually unnerved when modern SSDs die on us by asdfadsfgfdda on Tuesday December 11, 2018 @12:28PM (#18656931)
On modern MLC/TLC SSDs, read-only mode doesn't really exist. The NAND blocks must be re-programmed after a number of read accesses to mitigate read disturb. If anything, mounting read-only is probably a corner case, stressing the firmware's read disturb mitigation.

The tradeoff from worse read disturb characteristics is NAND that is 100x cheaper per GB than in 2000.

Re:Why I'm usually unnerved when modern SSDs die on us by vkuruthers on Tuesday December 11, 2018 @07:39PM (#18660374)
Do you have any more details on the re-programming that would be occurring on non SLC flash cells, even if mounted in read only mode? This is something I was always concerned about too.
Re:Why I'm usually unnerved when modern SSDs die on us by nickpsecurity on Tuesday December 11, 2018 @11:43PM (#18661454)
"We shouldn't have to do that."

I don't know about that. I know most managers in embedded software go cheap on engineers and software assurance on purpose to get bonuses and such. The hardware side makes me think you haven't studied deep, sub-micron hardware much. I started looking into it a few years ago or so, just reading the slides on lots of stuff even though not understanding much of it. They helpfully put a lot in lay terms, though, with lots of comparisons. To say the stuff gets harder every time you shrink to a smaller node is an understatement, esp for solid state.

If anything, the modern flash should probably be considered broken right as they ship out of the factory. If not, the process nodes after 90nm or so just keep having more and more ways for individual components to screw up or change behavior across same wafer. Some happen instantly by design. Some happen later with aging. The memory technologies are closer to that analog level of things than most with harder verification. The high-density flash on newer nodes uses less-reliable tech than most just to operate at that low cost. So, they add all kinds of firmware and hardware tricks to try to make it work for a period of time like it's a whole, functional unit of storage despite pieces of it misbehaving all throughout.

It's a nice, man-made miracle these techs even work at all. Those that last longer like you mentioned still exist. I'll add a comment with links to one type so you can compare price/storage/performance to these broken-by-design SSD's you use. I'll throw in another two that mention shrinking challenges so you can see what they face every time they have to upgrade or just deploy new designs in mixed-signal.

https://news.ycombinator.com/item?id=18528023

https://www.electronicdesign.com/digital-ics/understanding-2...

https://anysilicon.com/ic-design-impact-in-moving-from-28nm-...

Re:Why I'm usually unnerved when modern SSDs die on us by amatecha on Tuesday December 11, 2018 @07:35PM (#18660350)
Yeah agreed 100%, for my RAID mirror setup I use drives from distinct manufacturers for that exact reason -- I can presumably expect a different failure rate (hopefully). :)
Re:Why I'm usually unnerved when modern SSDs die on us by NikolaeVarius on Tuesday December 11, 2018 @10:50AM (#18656082)
Well to be fair, there is an entire layer of abstraction at the SSD controller level that does tons of black box magic. It allows the OS to treat the SSD as another other storage device without letting the OS know what is going on.

So the combination of non-moving parts (making it hard/impossible to debug via physical inspection) combined with a tons of wear leveling/misc magic can defo make it seem like SSD's are magical.

Re:Why I'm usually unnerved when modern SSDs die on us by Wowfunhappy on Tuesday December 11, 2018 @11:00AM (#18656163)
Is there a good reason why we use separate SSD controllers instead of letting the primary cpu handle it? The obvious reason is backwards compatibility, but as more of computing moves to SSDs, is this still relevant?

ZFS has shown that removing layers of abstraction with regard to storage can be beneficial.

Re:Why I'm usually unnerved when modern SSDs die on us by pjc50 on Tuesday December 11, 2018 @11:10AM (#18656270)
That adds a round of latency, and makes it pretty much impossible to boot off the drive. The blocks aren't in the same order in the Flash as they are presented by its interface, and one of the main jobs of the controller is to re-order them.

It would be an interesting product to have, a raw block API to a Flash device with all the temporary state stored on the host - but a hard one to sell, as it's not differentiated in any way.

Re:Why I'm usually unnerved when modern SSDs die on us by blattimwind on Tuesday December 11, 2018 @02:33PM (#18658107)
There are host-managed SMR drives, which share a number of characteristics with flash storage.
Re:Why I'm usually unnerved when modern SSDs die on us by marcosdumay on Tuesday December 11, 2018 @12:05PM (#18656731)
The controller makes it possible to get a standard bus and a pre-installed driver and use it to access any kind of memory from any manufacturer and any technology. It's the kind of convenience that makes people buy hardware - it's the kind of thing that made SATA and USB win. The alternative is that once in a while you plug a driver into your computer and it won't work.

Besides, I don't think manufacturers want to release the best practices for using their memory.

Re:Why I'm usually unnerved when modern SSDs die on us by selectodude on Tuesday December 11, 2018 @11:25AM (#18656383)
Apple uses their T2 security enclave as the SSD controller and it's caused quite a stir.
Re:Why I'm usually unnerved when modern SSDs die on us by joosters on Tuesday December 11, 2018 @12:16PM (#18656813)
I wonder if you can read any extra state out of the T2 as a result, e.g. more information on wear leveling, temporary read failures and so on, more than the standard SMART counters?
Re:Why I'm usually unnerved when modern SSDs die on us by Malic on Tuesday December 11, 2018 @12:21PM (#18656864)
Where can I read more about this?
Re:Why I'm usually unnerved when modern SSDs die on us by pixl97 on Tuesday December 11, 2018 @11:08AM (#18656248)
Re:Why I'm usually unnerved when modern SSDs die on us by hanikesn on Tuesday December 11, 2018 @10:55AM (#18656128)
The exact same thing happens hard drives. There's a ton of magic going on.
Re:Why I'm usually unnerved when modern SSDs die on us by NikolaeVarius on Tuesday December 11, 2018 @10:57AM (#18656143)
Re:Why I'm usually unnerved when modern SSDs die on us by relieferator on Tuesday December 11, 2018 @12:47PM (#18657106)
Thanks for this youtube link. In the case of recovering deleted files from SSD's that have the trim command enabled (Forensics Writeblocker was used,) the following drives have a low probability of recoverability: Crucial, Intel, and Samsung (3core controller). Whereas Seagate, Supertalent (parallel ATA to SATA bridge chip,) OCZ, and Patriot; files could be recovered. If the drive is quick-formatted, and if trim is enabled, the data is completely gone on the following drives: Crucial, Intel and Samsung. The trim state impacts the most whether or not the data can be recovered. You can check if trim is enabled with the command: fsutil behavior query DisableDeleteNotify - if your result is 0, it is enabled (default).
Re:Why I'm usually unnerved when modern SSDs die on us by gpvos on Tuesday December 11, 2018 @10:58AM (#18656149)
Yes, but most technical people seem to have a mental model that explains most failures pretty well. Not so for SSDs, apparently.
Re:Why I'm usually unnerved when modern SSDs die on us by ars on Tuesday December 11, 2018 @10:52AM (#18656099)
> Why can't you do the same with SSDs?

Because they don't die incrementally. Wtih a hard disk you'll get bad sectors, growing slowly over time. Or a head crash and then it's all dead.

What could cause an entire SSD to die at once? I would totally understand bad sectors, but the whole thing at once? Where it doesn't even try to read existing data?

Re:Why I'm usually unnerved when modern SSDs die on us by jclulow on Tuesday December 11, 2018 @11:10AM (#18656266)
One potential cause is poorly constructed drive firmware which fails to account for a minor failure of some kind and crashes. If that unaccounted minor failure is persistent, the firmware may crash constantly and then you'd be unable to access even theoretically good parts of the drive.

Firmware -- undebuggable, unobservable, unfixable software, jammed into your devices -- is the enemy.

Re:Why I'm usually unnerved when modern SSDs die on us by jug on Tuesday December 11, 2018 @11:19AM (#18656335)
Sounds like there is a business opportunist here for open firmware or even one running on the CPU especially for the enterprise? But maybe they won’t bother either despite valuable data and uptime, and just rather throw more redundant SSD’s at the problem.
Re:Why I'm usually unnerved when modern SSDs die on us by someguydave on Tuesday December 11, 2018 @05:33PM (#18659625)
You would need to convince the very small number of Flash memory fabricators to release their detailed specifications and known product weaknesses.
Re:Why I'm usually unnerved when modern SSDs die on us by ryanmercer on Tuesday December 11, 2018 @11:56AM (#18656660)
>What could cause an entire SSD to die at once?

A component failing? Electrical components fail. Sometimes it's a manufacturing defect, sometimes a design defect, sometimes something under or over-volted and it was enough to cause damage to any given component.

Could be an IC, could be a capacitor, could be a poorly laid trace. A poorly shielded RF source could even damage any number of components.

I mean, in theory a single high charge particle from that rare cosmic ray that reaches the surface of earth running full-steam-ahead through an IC could cause just the right amount of damage to make it fail although this would be an incredibly improbable scenario.

Same goes for HDDs, televisions, your clock radio, whatever.

Re:Why I'm usually unnerved when modern SSDs die on us by marcosdumay on Tuesday December 11, 2018 @12:08PM (#18656751)
> What could cause an entire SSD to die at once?

SSD has a problem in that almost every failure case looks more like a head-scratch than a bad sector.

Even the most local problem that is excessive writing on a sector can take the entire chip down.

Re:Why I'm usually unnerved when modern SSDs die on us by vokep on Wednesday December 12, 2018 @07:14PM (#18668831)
There's a single wire in all computers, for some reason they always just have to include it, which if severed will completely disable the computer, as in, it won't even turn on, not even try to turn on.

That wire is the one delivering power. Point being, while things are so complicated that you often can have a lot going on and still try things, there are still single points of failure that can never be fully covered.

Re:Why I'm usually unnerved when modern SSDs die on us by blincoln on Tuesday December 11, 2018 @11:41AM (#18656524)
<i>Why shouldn't it? Isn't it just hardware too?</i>

In a mechanical hard drive, there are moving parts which can wear out due to friction, etc.

SSDs are solid-state, so it seems like at least theoretically, it should be possible to build one that keeps working for decades. e.g. I have solid-state hardware from the 70s and 80s which still functions.

I've always been a little mystified as to why SSDs' data areas wear out for that reason,[1] but that's a whole separate issue. The author of the article is writing about sudden failure of the device as a whole.

There are only two explanations I've ever heard for short lifetimes in electronics as a general industry.[2]

For devices manufactured after about 2000 there's tin whiskers,[3] which began to be a problem because of RoHS requirements. I'm not sure if that applies here, though.

If the device includes electrolytic capacitors, my understanding is that those generally have a finite lifetime as well, and it can be fairly short if they're poorly-made.

I'm not a hardware expert, though, so I'd be interested in hearing about other factors, and I'm sure the author of the article would too.

[1] I've seen lots of writeups of how wear-leveling works, etc., but never a good physical explanation of what is actually wearing out over time.

[2] Obviously there are other factors for specific devices, or specific designs. E.g. maybe parts of a specific device break over time due to thermal expansion and contraction if the device wasn't engineered to handle that properly.

[3] https://nepp.nasa.gov/whisker/background/

Re:Why I'm usually unnerved when modern SSDs die on us by blattimwind on Tuesday December 11, 2018 @02:39PM (#18658168)
> I've seen lots of writeups of how wear-leveling works, etc., but never a good physical explanation of what is actually wearing out over time.

SSDs are flash are basically EEPROMs. In an EEPROM one bit is stored in a dual-gate MOSFET. One Gate is a normal gate, the other is floating, i.e. it just is a small conductive island. The information is stored by (quite literally) shooting electrons through the insulation of the floating gate into it. They're then trapped on the gate; if you turn the second gate on, the transistor conducts iff the floating gate is also turned on. This shooting action happens to damage the insulation, which at some point is degraded enough that it can't keep the electrons trapped on the floating gate. Hence the electrons leave, together with your information.

> For devices manufactured after about 2000 there's tin whiskers,[3] which began to be a problem because of RoHS requirements.

The solder joints themselves usually don't form whiskers, which mostly grow on pure tin-plated surfaces, e.g. the pins of components (which previously used lead). Using lead-free solder is mostly co-incident with whisker risk, not the cause of the majority of problems.

Re:Why I'm usually unnerved when modern SSDs die on us by 0db532a0 on Tuesday December 11, 2018 @03:48PM (#18658706)
The electrons aren't shot through the insulation. They end up in the floating gate by quantum tunneling. The insulation by its name is non-conductive, and must be so that the electrons stay put. The degradation happens through general thermal wear.
Re:Why I'm usually unnerved when modern SSDs die on us by userbinator on Tuesday December 11, 2018 @08:52PM (#18660724)
The exact wear mechanism doesn't really matter (you must be referring to "hot carrier injection") --- the point is that to record data, electrons are being forced through a material which gradually wears it out.
Re:Why I'm usually unnerved when modern SSDs die on us by 0db532a0 on Wednesday December 12, 2018 @04:36AM (#18662544)
That was also my point. I was just pointing out that the idea that electrons shoot through the insulation to end up in the floating gate was not true.
Re:Why I'm usually unnerved when modern SSDs die on us by Dylan16807 on Wednesday December 12, 2018 @05:21AM (#18662698)
Thermal wear? I thought it was mostly based on write cycles. How can that be 'thermal'?
Re:Why I'm usually unnerved when modern SSDs die on us by 0db532a0 on Wednesday December 12, 2018 @09:45AM (#18664298)
The write cycles are realised by moving electrons through a material. This creates heat, which leads to gradual degradation of the material.
Re:Why I'm usually unnerved when modern SSDs die on us by Dylan16807 on Wednesday December 12, 2018 @02:44PM (#18667097)
Does the individual gate get that much hotter than the rest of the device?

If I keep my SSD 40 degrees cooler with a freezer, will it last an indefinite number of write cycles?

Re:Why I'm usually unnerved when modern SSDs die on us by 0db532a0 on Wednesday December 12, 2018 @03:52PM (#18667654)
I just looked this up, and it turns out I was completely wrong about how the wear is caused. Apparently it’s due to a build-up of electrons remaining in the floating gate over time, and not general thermal wear. It turns out that heating can even reverse this wear. Explained in this link: https://arstechnica.com/science/2012/11/nand-flash-gets-bake...
Re:Why I'm usually unnerved when modern SSDs die on us by falcolas on Tuesday December 11, 2018 @12:00PM (#18656690)
On 1): It's because each cell is essentially a consumable with a limited number of state transfers. It's similar in that way to how a solid state accellerometer still has a moving part inside it that can break or wear out over time.

EDIT: More reasons and methods for failure of "solid state" circuits and systems: https://news.ycombinator.com/item?id=14765868

Re:Why I'm usually unnerved when modern SSDs die on us by blincoln on Tuesday December 11, 2018 @08:50PM (#18660717)
I understand that there is something that makes the cells stop working over time. blattimwind's reply is the first actual explanation I've ever seen of what that something might be.
Re:Why I'm usually unnerved when modern SSDs die on us by pbhjpbhj on Tuesday December 11, 2018 @11:50AM (#18656613)
On the subject of capacitor death. Is there a robotic device that can take in a circuit board and swap out all the [electrolytic] caps?
Re:Why I'm usually unnerved when modern SSDs die on us by sigstoat on Tuesday December 11, 2018 @12:48PM (#18657115)
i've never seen or heard of an automated board rework system. thinking about the steps and things i've had to do to manually rework boards, your machine would need not only be able to apply force to pull parts off of boards without damaging the board, but also be prepared to restore pads / through holes to usable states after desoldering parts, before new ones could go back in.

there's a reason companies don't repair circuit boards in consumer electronics.

Re:Why I'm usually unnerved when modern SSDs die on us by pbhjpbhj on Tuesday December 11, 2018 @04:01PM (#18658839)
I've done it myself, save restoring pads -- to me it's just the sort of fiddly, finnickety thing that it seems like robots sold be good at. And, they're not going to burn their fingers or run out of hands to hold things!
Re:Why I'm usually unnerved when modern SSDs die on us by PeterisP on Tuesday December 11, 2018 @09:49PM (#18661000)
Robots are good at repetitive tasks and economy of scale. Doing a tricky action for ten thousand boards is something that robots are good at; doing a different tricky action on each of a hundred boards is not.

It doesn't seem plausible to have a business case where you'd get be able to get a large quantity of identical boards (that are otherwise good!), replace caps on them, and be able to sell them for much more than you got them - i.e. that the boards haven't become obsolete in that time. If there's no mass production, there's not much use for automation.

Re:Why I'm usually unnerved when modern SSDs die on us by rasz on Tuesday December 11, 2018 @09:33PM (#18660923)
Yes, and it mostly speaks Ukrainian, or Chinese.
Re:Why I'm usually unnerved when modern SSDs die on us by gpvos on Tuesday December 11, 2018 @10:56AM (#18656138)
> Why can't we come up with an understanding of why it fails?

Indeed, that is the question that the article's author seems to be asking.

Re:Why I'm usually unnerved when modern SSDs die on us by Severian on Tuesday December 11, 2018 @11:00AM (#18656164)
Well, considering it's the cheapest consumer SSD they could afford, and then connecting it to a SAS port might be a whole story in itself.
Re:Why I'm usually unnerved when modern SSDs die on us by oasisbob on Tuesday December 11, 2018 @04:00PM (#18658811)
The post resonated with me because of a stupid bug I hit in a ca 2011 SSD. (Samsung?) After 100 power-on cycles, the drive would brick itself. Required a firmware update in time to avoid.

That defect doesn't strike me as being inherently related to SSD media, but really left a bad taste in my mouth with what might be going on in the development process to lead to such instability.

Re:Why I'm usually unnerved when modern SSDs die on us by rb808 on Tuesday December 11, 2018 @12:52PM (#18657154)
I think the difference is familiarity. We've been using hard drives for decades so kinda know what to expect. SSDs have only been widely used 5-10y, so we're all just getting used to know how they work and how they die.