We had a customer with regularly failing tape backups. CRC errors, verify pass failures, even failed writes, and so forth.
We replaced the tapes with new ones. Same issues.
We replaced the tape drive with a new one. Still the same problems.
We replaced the internal ribbon cable and the SCSI controller. No luck.
Firmware flashed everything. Didn't help.
New server chassis, wiped the OS and reinstalled everything from scratch. Changed the backup software just in case. The backups still failed!
Literally no part was the same. I went on site to start looking into things like the power cables, the UPS, or vibration issues. Basically were getting desperate and grasping at straws.
I was sitting down in an office, casually chatting with the IT guy while we were waiting for 5pm so we could reboot the server. He's leaning back in is office chair, and he casually picks up one of the tape cartridges and throws it up in the air and then catches it before it hits the ground. Just playing. Over and over.
I asked him if he does that a lot.
"Yes, it's fun!" he answered.
ಠ_ಠ
What was your companys role? Backup services/devices?
Finally I took all the parts out of the original computer and put them in a different chassis and it worked! Put them back in the old chassis and back to the old problem.
Eventually I noticed that there was an extra stand-off in the first computer case and it was shorting out the motherboard.
It was literally the chassis causing the problem.
Motherboard is bricked. Ring DELL for support. After going through the rigmarole of explaining what had happened and that we had a bricked motherboard, the person on the phone said "Have you tried taking out the CPU and rebooting?"
To avoid further delay in getting a replacement sent (we had 4 hour on-site at the time), we went through the motions. Not surprisingly, the motherboard was substantially bricked without a CPU.
The DELL engineer that came on-site was suitably amused.
Nothing worked. Finally, I removed the processor from the motherboard, looked at it, and reinstalled it. The computer booted right up and never had another problem. Weird.