Re:Outage Stories: The copy and paste outage
by kevin_nisbet on Wednesday August 17, 2022 @03:40PM
(#32500788)
Interesting. Yea we had a few of these due to I think it was vlan trunking protocol but I don't really remember. We had a cycle over 3 or 4 years where:
1. Configuring VLANs is hard, let's turn on the Cisco feature to sync VLANs 2. Network switch from lab or another network would be moved to production, with a higher revision index. All switches sync from the highest revision. 3. Outage happens because VLANs in lab are not same as production. 4. Investigation says vlan trunking protocol is dangerous, and should be disabled. 5. A year later, forget about the outage and repeat.
I think eventually there was a feature that sort of solved this problem, but I don't remember after all this time.
Finally after a few minutes,our head of IT/networking/everything finds the tech's personal cell phone, and calls him from his own personal cell phone, because VOIP is dead in the water now. Tech unplugs network device from network, within 5 min everything self-heals. Untold amount of fines and fees for not being able to handle trades during the outage.