The network team could not work it out. The vendor could not work it out. But one of the IT managers had an explanation: me. Firstly, it was due to an OpenVPN I installed on a server (with permission-as a stopgap measure so we could remotely access the “next-gen data centre” because the networking team was taking too long to get the real VPN installed and it was blocking other teams on the project.) The explanation didn’t make any technical sense: the VPN is just an application, nothing to do with the core routers; but he wasn’t technical enough to understand that. They told me to shut it down, so I did (even though doing so inconvenienced the project), and lo and behold, it made zero difference to the problem. Then, he apparently even suggested at a management meeting (I wasn’t there but I heard about it) that I was sneaking in to the data centre at night or on the weekends to sabotage things, and that was why the new routers didn’t work. Apparently they even asked campus security for my physical access logs, which revealed I hadn’t been doing any such thing.

Eventually, the vendor worked out the problem. When you install the router, there was a step you had to change the VRRP IDs to give every router a unique ID on the network. Clearly explained in the documentation, obviously essential, apparently our networking team didn’t read that part. You plug one new router in, everything is fine; plug the second one in, well it still has the OOTB default VRRP ID, so now two core routers on the campus network have the same VRRP ID, and all the other routers got confused, and the whole thing fell apart. Both our networking team and the vendor’s support team were so focused on chasing some obscure bug they didn’t see the basic config issue.