News for hackers. Stuff that matters.
Re:GoDaddy outage caused by corrupted router tables by nettdata on Tuesday September 11, 2012 @12:24PM (#4506321)
Never mind when you start adding fault-tolerance and failover setups/configs to the mix.

It gets really complex really fast, and even the slightest lapse in attention to extreme detail can be disastrous and a nightmare to troubleshoot.

For instance, I once had a router reboot for some unknown reason, and the firmware/config wasn't properly flashed, so it reverted to the last point release of the firmware. That was enough to cause the failover heartbeat to be constantly triggered, and the master/slave routers just kept failing over to each other and corrupted all the ARP caches. Makes for a lot of fun when things work and then stop working in an inconsistent manner.

There are no comments on this comment yet.