News for hackers. Stuff that matters.
Re:Did Pixar accidentally delete Toy Story 2 during producti... by PeterisP on Wednesday January 18, 2017 @09:58AM (#13426698)
This particular case is one that's hard to test - you'd restore the backup, look at it, and it would look fine; all the files are there, almost all of them have perfect content, and even the broken files are "mostly" ok.

As the linked article states, they restored the backup seemingly sucessfully, and it took two days of normal work until someone noticed that the restored backup is actually not complete. How would you notice that in backup testing that (presumably) shouldn't take thousands of man-hours to do?

Re:Did Pixar accidentally delete Toy Story 2 during producti... by hackuser on Wednesday January 18, 2017 @04:19PM (#13430381)
Good points. High-assurance can be very expensive in almost any area of IT. Speaking generally, when the asset is that valuable, the IT team should take responsibility for anticipating those problems - difficult, but not impossible. Sometimes you just have to roll up your sleeves and dig into the hard problems.

Speaking specifically, based on what you describe (neither of us is fully informed, of course), the solutions are easy and cheap: Verify the number of bytes restored, the numbers of files and directories restored, and verify checksums (or something similar) for individual files.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by PeterisP on Wednesday January 18, 2017 @05:59PM (#13431276)
The impression I got from the descriptions of that incident and especially the followup was that their main weakness was not technical, but organizational - their core business consisted on making, versioning and using a very, very large number of data assets that was very important to them, but they apparently didn't have any good process of (a) inventory of what assets they have or should have, and (b) who is responsible for each (set of) assets. Instead, the assets "simply" were there when everything was going okay, and just as simply weren't there without anyone noticing when it wasn't.

If they had even the most rudimentary tracking or inventory of those assets/artifacts, the same technical problems would cause a much simpler and faster business recovery; instead, circumstances forced them to inventory something that they (a) possibly didn't have and (b) didn't know if it needed to exist in the first place, and (c) in a hurry, without preparation or adequate tools or people for that.

IT couldn't and cannot fix that - implementing a process may need some support from IT for tooling or a bit of automation, but most of the fix would be by and for the non-IT owners/keepers of that data.