Re:Stratus: Servers that won’t quit – The 24 year running co...
by syntheticnature on Sunday January 29, 2017 @07:07PM
(#13517663)
"The Fail-Safe Theorem: When a Fail-Safe system fails, it fails by failing to fail safe."
Re:Stratus: Servers that won’t quit – The 24 year running co...
by bcbrown on Monday January 30, 2017 @05:22PM
(#13525627)
I love that book. Have you read https://www.amazon.com/Introduction-General-Systems-Thinking...? It's similar, but a bit more rigorous.
Re:Stratus: Servers that won’t quit – The 24 year running co...
by throwawayish on Monday January 30, 2017 @04:41AM
(#13519720)
> The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed.
This is the area were inverted Unix philosophy applies: Give me good news as well. A weekly mail, that backups (or the filer or RAID or whatever) works A-OK. If the mail stops coming, something broke.
Re:Stratus: Servers that won’t quit – The 24 year running co...
by hackuser on Sunday January 29, 2017 @04:45PM
(#13516893)
> The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted.
Agreed. Another moral: Don't believe anything works until you see it with your own eyes.
Re:Stratus: Servers that won’t quit – The 24 year running co...
by protomyth on Sunday January 29, 2017 @07:24PM
(#13517723)
Had a friend unplug the UPS to test it while I was still configuring the new machine. "Can't really be sure unless you test it"
Re:Stratus: Servers that won’t quit – The 24 year running co...
by throwawayish on Monday January 30, 2017 @04:43AM
(#13519725)
Especially true for backups. Make sure those aren't write-only!
Re:Stratus: Servers that won’t quit – The 24 year running co...
by shubb on Sunday January 29, 2017 @06:30PM
(#13517492)
Note to self, if need to socially engineer my way into the equipment room at a large company, pretend to be an IBM tech answering a phone home...
Re:Stratus: Servers that won’t quit – The 24 year running co...
by protomyth on Sunday January 29, 2017 @07:21PM
(#13517705)
Good luck with that, as we do check these things before we allow folks to get to the servers. IBM has been doing this for a fair bit and we aren't exactly blind idiots.
It would be interesting where I am since you skip local crime and go straight to federal offense. Kept me away from booze in high school since its not minor in possessions, its federal bootlegging. I can only imagine what trying to enter a server room nets a person.
One day, the shared mount on a ton of systems around the company just stopped working. The master node (these have internal redundancy at the network, controller, shelf, and disk level) had failed over to the redundant node, which after time had so many disks fail without being replace that it went down hard. They had to work with NetApp support (at great expense) to get it back up, which took a few days. The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed.
The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. The other lesson here is that such systems don't exist in isolation and are only as good as their weakest link, which in this case was a now-unreachable SMTP server. Lastly, although the LCD displays were trying their best to also signal this fault condition, people who would walk by these physical machines were used to this, so they didn't bother to actually read the errors. Do that at the peril of anything dependent on that system.