$ for dev in `ls /dev/sd?`; do echo $dev; sudo smartctl -a $dev | grep Load_Cycle_Count; done | cut -d " " -f 2,40
/dev/sda
Load_Cycle_Count 24
/dev/sdb
Load_Cycle_Count 24
/dev/sdc
Load_Cycle_Count 1947798
/dev/sdd
Load_Cycle_Count 1907706
sda and sdb are the samsungs and sdc and sdd are the WDs. I also just replaced a failing Samsung Green drive in another machine with a WD and it already has a Load_Cycle_Count in the 10000s. I guess I need to start avoiding Green WDs at least, maybe the Greens altogether.We recently had 3 servers have two drives each fail within hours of each other, with about two weeks between each of the 3 servers. These were 3 out of 4 servers that had been configured at the same time, with drives from the same delivery - clearly something had gone wrong.
Usually we try to drive types, but we didn't have enough suitable drives when we had to bring these up. Thankfully we do have everything replicated multiple times and very much specifically avoided replicating things from one of the new servers to another.
When we brought them back online we got a chance to juggle drives around, so now they're nicely mixed in case we get more failures.
For my private setup, I've gone with a mirror + rsync to a separate drive with an entirely different filesystem + Crashplan. Setups like that seems paranoid until you suffer a catastrophic loss or near loss...
My first big scare like that was a decade or so ago when we had a 10 or 12 drive array of IBM Deathstar (Deskstars) that started failing, one drive after the other, about a week apart, and the array just barely managed to rebuild... Particularly because it slowed the array down so much during the rebuild, that we were unable to complete a backup a day while running our service too, and taking downtime was unacceptable. So our backups lagged further and further behind while we waited for the next drive failure.. Those were some tense weeks.
What I am doing now as soon as I get unrecoverable errors from the drives is to replace them one at a time with whatever is the best cost/TB drive. Whenever all the drives have been upgraded I can resize the array to the new minimum drive size.
The number I have been finding to be high is Hardware_ECC_Recovered (values between 1036555546 and 2699460003). Not sure if that's normal. I've also had two 1.5TB drives now end up with unrecoverable sectors. RAID recovers from that just fine but I've been replacing them as it keeps reoccurring and is supposed to be a signal of failing disks. These 1.5TB drives are a 3+´years old and I've been thrashing them a bit lately. I'd have expected them to last longer though.