CenturyLink 911 outage was caused by a single network card sending bad packets

Re:CenturyLink 911 outage was caused by a single network car... by extrapickles on Sunday December 30, 2018 @01:21PM (#18790509)

My reading of the tea leaves is that the invalid packets where for some low level control protocol.

I’ve had invalid spanning tree (protocol typically used to prevent loops in networks) packets cause a trunk link to flap as the invalid packet made the switch think its only trunk link (the link to the rest of the network) was part of a loop and shut it down. When the link went down, it could no longer get the bad packets, so after a delay it would enable the trunk again and get the invalid packets again.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by jandrese on Sunday December 30, 2018 @02:00PM (#18790687)

I've seen plenty of networks where there is a packet storm, then STP disables the link and fixes it, then 30 seconds later STP re-enables the link and the packet storm resumes...

I have a very low opinion of STP. I'm also not a fan of the fad to have very flat networks where there are very few routers and instead everything is switches in one gigantic subnet. Packet storms are notoriously difficult to track down on big networks.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by drcross on Sunday December 30, 2018 @07:50PM (#18792272)

>I have a very low opinion of STP.

That's not a very radical opinion in network engineering circles. No one ever liked it due to it's non forwarding links for loop prevention but it was good enough to work until we discovered it's successor.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by oarsinsync on Monday December 31, 2018 @07:16AM (#18794187)

> until we discovered it's successor

Which is?

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by insaneirish on Monday December 31, 2018 @08:30AM (#18794391)

Ensuring all layer 2 domains are loop free by design and moving all redundancy to layer 3 and routing protocols.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by oarsinsync on Wednesday January 02, 2019 @03:00AM (#18804852)

> Ensuring all layer 2 domains are loop free by design

Impossible, unfortunately, or we wouldn't need STP.

> and moving all redundancy to layer 3 and routing protocols.

Not always possible either, sadly. Works well for new deployments, can difficult to retrofit into older deployments depending on the scale and applications involved.

[ Parent | on HN ]