Re:ST3000DM001
by matclayton on Monday June 07, 2021 @04:34AM
(#27420343)
That serial number sends shivers down my spine. I also used to work on this system, and still do! At one point seagate admitted that the firmware was faulty. It would periodically stop responding under load, and cause the raid controller to think the drive had failed, causing it to remove it from the array.
I’ve never seen this posted publicly by them, but I’m fairly sure a revised firmware was offered. We deemed it too risky to upgrade the drives one at a time, so built a new cluster on Western Digital drives.
These things always happen in the weekend don't they? When I managed a large network of servers, it seems that all the outages happened during parties and/or the weekend.
Issues seem to happen as you implement the long desired upgrade/fix/repair too. Things hold on for years then crap out right at the final moment.
This makes sense though doesn’t it? Rebuilding an array is a very heavy load op, and putting components that have been around for a while under heavy load seems like a good and fast way to expose failing parts.
Re:ST3000DM001
by spookthesunset on Monday June 07, 2021 @01:56PM
(#27425769)
> These things always happen in the weekend don't they?
Thats because servers know.... oh they know. Tuesday morning when you get into the office? Naw... that's too easy. Let's shit the bed on Christmas morning at 3:49am. That will let those stupid humans know who is boss around here.
1) At one point we were experiencing disk failures often enough that we wrote a cronjob to detect drive failures via smartctl and automatically send an email to our hosting company requesting them replace the drive. This saved engineering time, and more importantly reduced time to drive replacement, because
2) On at least one occasion, we had the third drive in a RAID6 array fail before we had rebuilt from the initial failure, leading to loss of the array. We think the increased load of the rebuild increased the chance of subsequent failures. Needless to say recovering from this destroyed all plans (and sleep) for the weekend it happened.