News for hackers. Stuff that matters.
Re:SSD will fail at 40k power-on hours (2021) by userbinator on Sunday July 10, 2022 @05:17PM (#32049143)
40000 (or even 40960) seems an odd number to fail at. 64k or 32k would make the cause pretty obvious, but 40000 doesn't seem all that round in binary. Perhaps a 12-bit counter incrementing every 10h? This is puzzling.

Of course, I am also entertaining the possibility that no one thought they would be in use for this long, which would certainly be evidence of planned obsolescence.

Re:SSD will fail at 40k power-on hours (2021) by twawaaay on Sunday July 10, 2022 @05:23PM (#32049193)
Very strange understanding of the word "evidence".

No sane SSD manufacturer would do such thing on purpose. You do it and you loose business, that's it.

The simplest explanation is that somebody made an honest engineering mistake.

Re:SSD will fail at 40k power-on hours (2021) by bayindirh on Sunday July 10, 2022 @05:57PM (#32049500)
When you purchase a server (fleet), you get a long warranty with it. Generally 3 to 5 years. So you expect this fleet to stay in service for <=5 years mostly.

Unless you burn through your SSDs, you're very unlikely to hit this event.

When these servers' continue to be used and disks all start to fail at the same time, this will obviously stink.

The bathtub curve is not like this. You can feel that.

Re:SSD will fail at 40k power-on hours (2021) by toast0 on Sunday July 10, 2022 @08:36PM (#32050835)
40k hours is a little more than 4.5 years. These drives deterministically fail at that uptime (unless firmware is updated) and most servers are on 24x7, so if you run your servers for 5 years, it's highly likely you'd run into this. If you run your SSDs hard and they fail early as a result, then you'd be spared from this mass death. Or if you use three year leases and replace on a schedule.

Now more than ever, five year old server hardware isn't that far behind the curve unless you're on the bleeding edge. I've been looking for bottom of the barrel hosting lately, and there's lots of dedicated servers available with 10+ year old cpus, and probably most of the rest of the machine is a similar age.

Re:SSD will fail at 40k power-on hours (2021) by jstrong on Monday July 11, 2022 @09:40AM (#32055980)
you have any recs for lower-end bare metal providers?
Re:SSD will fail at 40k power-on hours (2021) by toast0 on Tuesday July 12, 2022 @02:21AM (#32066296)
I haven't used them yet, but take a look at dedicated server offers at LowEndTalk and Web Hosting Talk. Obviously, there's some offers that are sketchier than others, and a lot of resale, but if your needs are low, there's some neat stuff.
Re:SSD will fail at 40k power-on hours (2021) by fartcannon on Sunday July 10, 2022 @05:43PM (#32049383)
Given the power dynamic between a single customer and large corporations, the smart thing to do is to assume malice until prove otherwise. This puts the onus on the corporations and, if we're lucky, creates an environment where they compete with each other to be seen as the most honest. The worst thing that happens is the single customer has to buy an SSD from someone they don't trust.

If we do the opposite, as you say, and assume everything is an honest mistake, that puts pressure on the single customer to prove that the organization with a huge marketing budget is doing something wrong. In this situation, the worst thing that happens is we all get taken advantage of.

Our collective distrust is the only power we have against massive marketing/PR budgets. It doesn't have to be angry, or sour, or cranky, we just collectively need to not take their word until we have a reason to do so.

Re:SSD will fail at 40k power-on hours (2021) by charcircuit on Sunday July 10, 2022 @05:49PM (#32049441)
Are you seriously saying that by default we should believe they intentionally planned to cause their customers to lose all of their data?
Re:SSD will fail at 40k power-on hours (2021) by helionsantos on Monday July 11, 2022 @04:41AM (#32053675)
Considering immoral practices adopted by corporations, such as vendor lock-in, use of slave work (directly or indirectly), law bending for its own interests, supporting and conducting biased research towards its own interests among others, I would say that is quite sensible to believe it. Big corporations, per se, are not evil entities. The people running them might or might not be, and when you have evil/immoral people running things, unless there are good control measures in place, they might take bad decisions.
Re:SSD will fail at 40k power-on hours (2021) by bayindirh on Sunday July 10, 2022 @06:01PM (#32049536)
If a spinning rust can run for ~8 years without any problems,a consumer SSD can hit beyond 40K hours reliably, and everything is checked and tested tens of times because of the complexity of flash storage, I'd get suspicious too.

Also, enterprise drives get firmware updates (regardless of spinning or not), and this firmware is automatically applied via RAID controller, so it could be remedied easily before it got this big if it's an actual error.

Re:SSD will fail at 40k power-on hours (2021) by alliao on Sunday July 10, 2022 @05:54PM (#32049476)
planned obsolescence is quite a thing...?
Re:SSD will fail at 40k power-on hours (2021) by dtjb on Sunday July 10, 2022 @06:29PM (#32049805)
In some cases, but a product must fulfill its core purpose. If a SSD intentionally dumped data and self destructed at a set time, that would be disastrous for the brand. Same way a car doesn't adopt planned obsolescence by blowing up after 200k miles.
Re:SSD will fail at 40k power-on hours (2021) by cmeacham98 on Sunday July 10, 2022 @10:40PM (#32051579)
> If a SSD intentionally dumped data and self destructed at a set time, that would be disastrous for the brand.

Other than "intentionally" (which we cannot know and makes no difference to whether you lose your data or not) that is literally what these SSDs are doing, and no SSD brand has been destroyed over it.

Re:SSD will fail at 40k power-on hours (2021) by landemva on Sunday July 10, 2022 @08:04PM (#32050621)
You are not a used car afficionado?

'This insulation prematurely disintegrates under normal use causing the wires it is designed to protect and insulate, to short causing many problems.'

http://www.mercedesdefects.com/2008/04/wire-harness-defect.h...

Re:SSD will fail at 40k power-on hours (2021) by bmicraft on Monday July 11, 2022 @03:54AM (#32053418)
What more could a manufacturer do to be "disastrous for the brand" than literally build an ssd that stops working after 40kh? Because this does not seem to qualify for you
Re:SSD will fail at 40k power-on hours (2021) by justinsaccount on Sunday July 10, 2022 @05:23PM (#32049196)
Someone pointed out on the other thread that it could be 2^57 nanoseconds:

  >>> 2**57/10**9/3600
  40031.996687737745
Re:SSD will fail at 40k power-on hours (2021) by AaronFriel on Sunday July 10, 2022 @05:30PM (#32049264)
If it were 53, I'd wonder "are they storing the time in the integer part of a double precision float?" That wouldn't go negative, it'd just start absorbing increments without changing the value.

Though that might cause a divide by zero?

What could cause unexpected behavior at 57 bits?

Perhaps storing fractions of an hour, like incrementing it every 1/16th of an hour and calculating a relative rate of change, causing a divide by zero?

Re:SSD will fail at 40k power-on hours (2021) by jonas21 on Sunday July 10, 2022 @06:10PM (#32049616)
My overactive imagination thinks it went something like this:

Engineer A: Gee, I need to store a few flags with each block, but there's nowhere to put them. Ah! We're storing timestamps as 64-bit microseconds. I can borrow a few of those bits and there'll still be enough to go for thousands of years without overflowing.

Engineer B: Gee, our SSDs are getting so fast, soon we'll be able to hit 1M writes/sec. But we're storing timestamps as microseconds. How can we generate unique timestamps for each write? Ah! I'll switch to nanoseconds. It's a good thing we have plenty of space in this 64-bit int.

BOOM!

Re:SSD will fail at 40k power-on hours (2021) by danielheath on Sunday July 10, 2022 @05:57PM (#32049496)
Packing a type flag into the upper bits of a 64 bit value is a reasonably common optimisation in dynamic language implementations (because it lets you use unboxed number arithmetic).
Re:SSD will fail at 40k power-on hours (2021) by KMag on Sunday July 10, 2022 @10:39PM (#32051571)
Or sometimes the lower bits, as at least used to be the case for integers in v8. (Also OCaml, but that's not dynamically typed. It simplifies the garbage collector to at least some times not require a pointer map for each type, just a flag in the object header to indicate if it contains any pointers, and then everything that isn't ints or pointers needs to be boxed.)
Re:SSD will fail at 40k power-on hours (2021) by mkl on Sunday July 10, 2022 @05:52PM (#32049463)
Do embedded CPUs like the one in an SSD have floating point units? It seems more likely to me that the upper bits in a 64 bit integer counter were used for something else.
Re:SSD will fail at 40k power-on hours (2021) by R0b0t1 on Monday July 11, 2022 @06:25PM (#32062859)
I think it is more likely they shifted a power of two over implicitly by a base 10 place value instead of a binary one. Or multiplied by 10. Unsure why. But, seems simpler.

52 is notable as 2^4 + 2^3 = 24, 24 + 24 = 48, 48 + 2^2 = 52. But 57?

Re:SSD will fail at 40k power-on hours (2021) by tyingq on Sunday July 10, 2022 @07:01PM (#32050076)
From a related issue with a different vendor:

"The fault fixed by the Dell EMC firmware concerns an Assert function which had a bad check to validate the value of a circular buffer’s index value. Instead of checking the maximum value as N, it checked for N-1. The fix corrects the assert check to use the maximum value as N."

https://www.anandtech.com/show/15673/dell-hpe-updates-for-40...

Why the MAX value would be in an circular buffer, or what was being stored in N-1? No idea.

Re:SSD will fail at 40k power-on hours (2021) by KMag on Sunday July 10, 2022 @10:36PM (#32051560)
From my reading, it's checking the maximum index into the circular buffer. That is, when it hits the end of the circular buffer, there's an assertion to check that they're properly wrapping the index back to the start of the buffer, but the assertion has an off-by-one error.

I presume you find a lot of circular buffers in SSD firmware, for wear-leveling reasons. Samsung's NILFS and NILFS2 are structured as circular buffer append-only logs, at least partly to avoid trusting the firmware wear-leveling.

Re:SSD will fail at 40k power-on hours (2021) by userbinator on Sunday July 10, 2022 @07:24PM (#32050267)
circular buffer

The infamous Seagate firmware bug was due to the same thing.

Re:SSD will fail at 40k power-on hours (2021) by chaboud on Monday July 11, 2022 @06:26PM (#32062874)
2^57 nanoseconds is ~40032 hours. I wouldn't be surprised if someone out there was counting intervals in a 64-bit value and masking off some of the higher bits for flagging.

Any time I see these sorts of issues (odometers that kill themselves, for example) I think of smaller units at higher bit depths. That's not the only way to get to this kind of concern, but it's a way that pretty-darn-competent engineers can leave ticking time bombs due to estimation failures.

Always check types for overflow and/or precision loss. Always.

Re:SSD will fail at 40k power-on hours (2021) by R0b0t1 on Monday July 11, 2022 @06:21PM (#32062822)
It's a power of two shifted by a decimal place value instead of a binary one. Unsure why.