A quick self test every day for all disks, and a long (i.e. full read) self test once a week.
The RAID is then checked on top of that one a month (although that slows things down a bit).
A better plan is to light up your disaster recovery plan weekly, and while the DR system is handling the load, scrub to your hearts content on the down system.
Depending on the cost of your hardware vs the cost of your labor vs the cost of downtime, dual servers, one flagged as production and one flagged as development, alternate flags every weekend, might work out. You'll hear lots of bragging about that not being possible because the hardware is too expensive, not so much bragging about labor cost and downtime cost. I worked at financial services corp about two decades ago where downtime was supposedly in excess of $1M/hr. They had triple mainframes set up, basically three machine rooms inside the machine room.
The combination of BMS and disk scrubbing at the RAID level should handle almost all of the issues that are pointed by the original post.
Though RAID scrubs can and do take a long time to complete, depending on the performance impact that you are willing to suffer on a continuous basis it can take a week or two to perform proper scrubbing.
Proper scrubbing would include not just reading the RAID chunk on a disk but to also read the other associated chunks from the other disks and verify that the parity is still intact. In RAID5 you will not be able to recover if the parity is bad as you won't know what chunk has gone bad.
I've been coding such systems for a while now and as a shameless plug would point to http://disksurvey.com/blog/ if there are things of interest I'd be happy to take requests and write about them as well.
The article assumes no scrubbing, which is a stupid thing to run without, as detailed from the article. So it's basically "why pointing a gun at your foot and pulling the trigger is bad", "because you're going to shoot yourself in the foot".
daily_status_zfs_enable="YES" daily_scrub_zfs_enable="YES" daily_scrub_zfs_default_threshold="6" # in days
and it will scrub the pools every 6 days (and send you a report in the daily run output).
If you run "camcontrol identify ada0" (or whatever your device is) you can find out before it is too late:
sector size logical 512, physical 512, offset 0
This is from a lucky drive of course :)
I don't have a Linux box available right now but maybe "hdparm -I" does something similar: "request identification info directly from the drive".
Logical Sector size: 512 bytes
Physical Sector size: 4096 bytes
I'm guessing that's not very nice. My ZFS pool was created with ashift 9 (this is 2^9=512 bytes), when it should be 12 (2^12=4096). I will have to copy everything off and back on again.For everyone who wants to check, and because I couldn't find info on it, run:
zdb | grep ashift
And see if it's 9 or 12.
But from the beginning of TFA, after reading this:
"Bad blocks. Two of them. However, as the blocks aren't anywhere near the active volumes they go largely undetected."
The FIRST* thing that came to my mind was: "What!? Isn't that a long-solved problem!? Aren't disks / controllers / RAID setups much better now at detecting such problem right away".
I've got a huge issue with the "largely undetected". I may, at one point, need storage for a gig I'm working on. And I certainly don't want stuff problems like that to go "largely undetected".
So quickly skipping most of the article and going to the comments:
"It's worth pointing out that many hardware RAID controller support a periodic "scrubbing" operation ("Patrol Read" on Dell PERC controllers, "background scrub" on HP Smart Array controllers), and some software RAID implementations can do something similar (the "check" functionality in Linux md-style software RAID, for example). Running these kinds of tasks periodically will help maintain the "health" of your RAID arrays by forcing disks to perform block-level relocations for media defects and to accurately report uncorrectable errors up to the RAID controller or software RAID in a timely fashion."
To which the author of TFA himself replies:
"Yes, that is something I should have made clearer. This is the very reason that RAID systems have background processes that scan all the blocks."
Which leaves me all a bit confused about TFA, despite all the shiny graphs.
Basically, I don't really understand the premises of "bad blocks going largely undetected" in 2013...