News for hackers. Stuff that matters.
Re:Replacing a silently failing disk in a ZFS pool by toast0 on Wednesday July 03, 2019 @10:49AM (#20344724)
As marcosdumay says, smart should handle this, but you do have to look.

Iffy sectors get marked as Pending, Reallocated sectors are also marked. There's some smart stats for seek times too, I believe (but those weren't as clearly associated with failure as sector counts).

There is a danger that manufacturers will avoid acknowledging problems in smart, but so far, I haven't heard about that happening too often. There are certainly some drives that are dead or dying with fine smart values, but it seems rare, and there's often a not obvious dependency where a drive that is very unhealthy may not be able to record new smart values.

Re:Replacing a silently failing disk in a ZFS pool by ianhowson on Wednesday July 03, 2019 @05:22PM (#20348942)
In my experience, SMART doesn't detect anything of value. 95% of my disk failures are:

- Seek time gets very high (>80%). - Drive drops dead and won't respond

SMART doesn't detect either of these.

Setting an alert on seek times > 30ms has been by far the best predictor of drive failure. If the seek times go over ~400ms then other components start complaining about slow/missing I/O, but by that point you've already gone crazy because your array is unusably slow.

If a drive develops bad sectors SMART will report it, but that's rare enough that I don't bother watching for it. It's a non-event with ZFS because of the checksumming. These drives never have one random error; they start raising hundreds in the space of minutes, and that's the moment you replace the drive.

Re:Replacing a silently failing disk in a ZFS pool by dusted on Thursday July 04, 2019 @09:21AM (#20354342)
Smart shouldn't be used for this, the sata transport layer should be updated to allow indicating it, syscalls should pass this information along, so filesystems can react to it.. otherwise we'd need a new smart field, as the semantics of the existing fields are insufficiently described, and reacting to them would require adding a smart-query to every filesystem call that touches disk.

thing is, "iffy" depends on what firmware decides is iffy.. Maybe a firmware relocates a sector the very first time it fails to read, maybe it relocates after try 200. What we want is to be notified, when we request a sector, if reading it was 100% painless. Depending on your demands for reliability, maybe it's okay to re-read a sector a few times, and still not reallocate it, but maybe your filesystem would like to gather these stats from the disks anyway.