CenturyLink 911 outage was caused by a single network card sending bad packets

Re:CenturyLink 911 outage was caused by a single network car... by Jach on Sunday December 30, 2018 @04:38PM (#18791496)

From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server:

    lookupByCity(...) {
        ....
        try {
            conn = connectionPool.getConnection();
            stmt = conn.createStatement();
            ...
         } finally {
             if (stmt != null) {
                 stmt.close();
             }
             if (conn != null) {
                 conn.close();
             }
         }
    }

close() can throw, and in the circumstance of the outage it did for stmt, leading to the connection not getting closed and eventually the pool being exhausted with every thread blocked waiting for a connection. It's an interesting chain of failures, arguably the presence of such a chain is the real root cause, rather than the unhandled sql exception.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by bwduncan on Sunday December 30, 2018 @07:46PM (#18792258)

the presence of such a chain is the real root cause, rather than the unhandled sql exception.

This is really interesting and something which bugs me about root cause analysis and it's a neat coincidence that this has been quoted relative to an aviation incident.

In aviation, incidents and accidents are investigated with the understanding that there is never a single cause of an accident. It's known as the swiss cheese model. All the holes in the swiss cheese have to line up for something to go wrong. Even in a seemingly simple "pilot error" accident, there are years of initial and recurrent training factors, ergonomic and human factors and so on which all lead to the event. It's exceedingly rare for a single "root cause" to be the whole story.

Medicine is starting to adopt techniques learned from aviation like checklists, crew resource management and no-blame, swiss-cheese accident investigations. I am hopeful that the software industry will take similar lessons over the next decade or so.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by Rapzid on Sunday December 30, 2018 @11:17PM (#18792983)

> like checklists, crew resource management and no-blame, swiss-cheese accident investigations. I am hopeful that the software industry will take similar lessons over the next decade or so.

The software industry that programs space craft?

The software industry is vast and not every system involves copious amounts of human decision making. Often the idea of the root system cause, and the root process cause(software construction, operations, etc) cause is separable.

I would say that aviation is almost inverted in the that regard compared to booking systems, banking systems, and most of what software engineers are exposed to. A person can not fly from Dallas to Chicago without many, many human decisions being involved. However, a packet traveling from Dallas to Chicago involves nearly zero new human interactions.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by throw0u1t on Sunday December 30, 2018 @07:38PM (#18792222)

I know people like to hate on errors as values as in Go, but I think exceptions are worse when it comes to unexpected side effects and this is a prime example!

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by josephg on Sunday December 30, 2018 @08:37PM (#18792494)

Yes; although in GC’ed languages like Go and JS it’s still very easy to leak OS level resources like file handles because you still need to remember to close() them. (Although go’s defer blocks are a fantastic assist here).

This is one area Rust really excels - the same mechanism for making sure memory gets cleaned up also automatically closes network sockets and file descriptors when they go out of scope. Even in the case of errors it’s impossible to forget to clean up. That entire finally block is unnecessary in rust.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by throw0u1t on Sunday December 30, 2018 @08:47PM (#18792524)

You're talking about the Drop trait?

I'm learning Rust coming from Go. It looks cool, but it also concerns me how most data structures in the stdlib use unsafe blocks to defeat the borrow checker. This is not the point of Rust, I would have thought?!

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by steveklabnik on Sunday December 30, 2018 @09:16PM (#18792613)

One of the criteria for belonging in the standard library is “needs a lot of unsafe to implement”, so the standard library has more unsafe code than your average codebase.

Beyond that, to some degree, it is the point of Rust: limit unsafe things so that you can reason about them more easily. The CPU is inherently not safe, so it has to exist on some level. rust gives you tools to manage this.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by cpeterso on Sunday December 30, 2018 @05:41PM (#18791780)

"Release It" is an awesome book. I contributed a couple FindBug Java checkers to warn about some of the problems the book describes.

The case you mention above might have been prevented by using checked Java exceptions. Our programming languages and tools could be doing a lot more to catch these problems at compile time or make them impossible by language or API design.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by Deinumite on Sunday December 30, 2018 @06:33PM (#18791992)

I think that was taken into account when the "try with resources" statements were added to Java.

As you said it would be nice if the languages could make these situations impossible.

I think its a bit of a chicken and egg problem because some language issues don't come to light until people are using it and it is too late.

[ Parent | on HN ]

Re:CenturyLink 911 outage was caused by a single network car... by mey on Sunday December 30, 2018 @07:41PM (#18792235)

My favorite is constructors of things that taken input/output streams that themselves can throw. It becomes very verbose to ensure that the unwinding happens correctly.

[ Parent | on HN ]