News for hackers. Stuff that matters.
Re:A coding error caused Rogers outage that left millions wi... by ajross on Monday July 25, 2022 @08:08PM (#32232648)
In general, you can't do any kind of automated rollback when the network goes down (same deal for DNS, which is why I mentioned it -- similar datacenter-wide failures have been due to stuff like stampeding initialization herds, etc...). I get what you're saying, I'm just pointing out that it's a little naive. No matter how high you make your stack of abstractions, there's still a human being at the top of the stack with fat fingers.
Re:A coding error caused Rogers outage that left millions wi... by zeroimpl on Monday July 25, 2022 @08:39PM (#32232883)
Sure but that’s why I think the take-away here is to design the system such that you can do such automated rollbacks. For example, the devices should also be connected to some sort of isolated management network. Granted, pushing out a config to the routers could also break their connections to said management network, but it is less likely anyways.