Re:What SMART Stats Tell Us About Hard Drives
by Retric on Thursday October 06, 2016 @06:04PM
(#12656523)
Thresholds are often useful with these kinds of stats. Aka a drive moving one sector might mean nothing, but moving 30 in a week could be great predictor. Further they only have 70k drives across a range of product lines so what predicts drive X failing very well might say little about drive Y.
PS: Rememebr all RAM gets bit flit errors over time. Which is one of the reasons rebooting is often so useful, but also means one off errors are often meaningless.
Re:What SMART Stats Tell Us About Hard Drives
by sn on Thursday October 06, 2016 @07:50PM
(#12657091)
ECC ram will either correct the error or can raise a non maskable interrupt if it can't be corrected.
Re:What SMART Stats Tell Us About Hard Drives
by Retric on Thursday October 06, 2016 @08:24PM
(#12657262)
ECC vastly lowers, but does not remove this problem. You can for example get bit flip errors while doing the checksum.
As a personal user it's a non issue but scale things to ~70k devices and you get ~1.7 million device hours per day.
Re:What SMART Stats Tell Us About Hard Drives
by sn on Thursday October 06, 2016 @08:55PM
(#12657378)
Fair enough. Your original comment did not make this distinction.
I dug and found https://www.fiala.me/pubs/papers/sc12-redmpi.pdf the title of which is "Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing."
They state that for a cray there was a double bit flip about 1x/day for 75k modules. To (probably incorrectly) extrapolate, if you have a server with 16 modules that would be equivalent to a single double failure about once every 13 years.
Re:What SMART Stats Tell Us About Hard Drives
by joosters on Friday October 07, 2016 @06:18AM
(#12659481)
It's unfair to blame SMART stats for this. On their own, SMART does not declare a drive 'healthy' or 'unhealthy', it's up to the user to decide. Each stat on its own seems very specific, e.g. Number of reallocated sectors. I'm not sure how you think this is 'poor specificity', the stats are measuring one thing exactly, you can't get more specific than that!
Re:What SMART Stats Tell Us About Hard Drives
by fnj on Friday October 07, 2016 @06:49AM
(#12659595)
Actually, it does declare. Direct quote from smartctl output: "SMART overall-health self-assessment test result: PASSED"
Re:What SMART Stats Tell Us About Hard Drives
by devonkim on Thursday October 06, 2016 @10:07PM
(#12657636)
SMART stats are only one source of data though that can be predictors of drive failures. Hard drives can experience latency spikes and erratic performance compared to peers before SMART records problems, for example. This is part of why predictive monitoring can be so difficult - the data you didn't know you needed is probably among the ones you didn't ingest into your metrics, and you can't dump literally everything in /proc every other second to your metrics system without sacrificing some CPU or network bandwidth either.
Re:What SMART Stats Tell Us About Hard Drives
by andy4blaze on Thursday October 06, 2016 @05:24PM
(#12656241)
Nice analysis.
Take Backblaze's 0.01% for a group of four failures. That's replacing an extra 100 drives per million, at random, and only getting the benefit of correctly predicting failures 10.4% of the time.
This is great data to have.