CenturyLink 911 outage was caused by a single network card sending bad packets

Re:CenturyLink 911 outage was caused by a single network car... by scurvy on Sunday December 30, 2018 @11:51PM (#18793079)

It's hard to work on network problems when the management network is also having problems. That requires dispatching people locally to sites to see what's going on. You're blind in both eyes, not just one.

Look, I'm not defending them or their choices. I'm just saying that this more more nuanced than "They suck they should have known better or spent more money." When you run a network as large as this and with as many "legacy" technologies as they do, things aren't very cut and dry.

NTT probably has the best managed global network with their extensive use of SDN. They run a tight ship and everything goes through their automation frameworks. This requires a huge investment in R&D. Even then, it doesn't cover 100% of their network because "there's that T320 for that customer in Chicago, and the stuff up in Michigan for Dorian's T1." Multiply that times 30 or so to consider size differences between NTT and CenturyLink....and yeah, stuff happens.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by wpietri on Monday December 31, 2018 @11:02AM (#18795088)

I agree it's hard. Anything interesting is.

But your last paragraph specifically describes technical debt as part of the problem. If even the best org (NTT) has technical debt and CenturyLink isn't the best org, then I think it's safe to suspect that a) CenturyLink has significant technical debt, and b) they have underinvested in that technical debt compared with industry best practices.

[ Parent | on HN ]