They had a backup server. Misplaced automation is the villain. Makes sense to automate a task that you do 100+ times a day, but backups are generally run once a day, and are recovered maybe once a year.

My backup system involves a script with a bunch of copy commands that I can see flying by in the console as they work.