Re:Did Pixar accidentally delete Toy Story 2 during producti...
by greedo on Tuesday January 17, 2017 @12:23PM
(#13419258)
We had a proper backup system in place at my company. Backups were replicated to an identical system at a remote site, and we periodically validated random backups. Our audit team also spot checked our backups to ensure that all servers were being backed up. Then we had a management change; ditched our system for a more expensive solution, and basically told our application owners that backups were a nice thing to have, but they shouldn't really count on them. The person in charge of backups was assigned other tasks and encouraged not to spend too much time on managing the backup system. With over 1000 servers, it's a matter of time until we have an issue that leads to a CTJ event. Unfortunately, the backup admin will probably be the one facing Jesus, not the management staff...
An executive usually requires a "Come to Jesus" moment like this one, where the entire company teeters on the precipice due to lax backup or security policies, to really have the importance impressed upon him or her. At that point, they are generally much more supportive, though sadly, this too can start to fade if the sysadmins do their job too well.
I don't want anyone to come away thinking that most companies have solved these problems. It's definitely not the case, even in large, established companies. Security and backups continue to get little attention until it's too late.
We really need to call a celebrity in MBA circles and get that person to run a seminar meant to scare the pants off the execs.