News for hackers. Stuff that matters.
Re:Level 3 Global Outage by phkahler on Sunday August 30, 2020 @12:35PM (#24323904)
Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change.

This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

Re:Level 3 Global Outage by jacquesm on Sunday August 30, 2020 @06:32PM (#24326521)
> Normal testing was bypassed - per management request after a small code change.

That lesson will really never be learned. This happens on a daily basis all over the planet with people who have not been bitten - yet.

Re:Level 3 Global Outage by cortesoft on Sunday August 30, 2020 @07:18PM (#24326856)
It isn't learned because 99% of the time, it works fine and nothing bad happens.

We are very bad at avoiding these sorts of rare, catastrophic events.

Re:Level 3 Global Outage by eru on Sunday August 30, 2020 @10:13PM (#24327900)
That's why the most reliable way to instil this lesson is to instil it into our tools. Automate as much testing as possible, so that bypassing the tests becomes more work than running them.
Re:Level 3 Global Outage by smaudet on Monday August 31, 2020 @07:46AM (#24330435)
I disagree, it's in part a people problem - more draconian test suites just make developers more inclined to cheat and they tend to write tests which are not valid or just get the tool passing...

It's more important to visually model and test than to enforce some arbitrary set of rules that don't apply universally - then you have at least the visual impetus of 'this is wrong' or 'I need to test this right'.

A lot of time is spent visually testing UIs and yet these same people struggle with testing the code that matters...

Re:Level 3 Global Outage by monkpit on Monday August 31, 2020 @11:05AM (#24332183)
Until a manager is told about how hard the automation makes it to accomplish their goal...
Re:Level 3 Global Outage by eru on Tuesday September 01, 2020 @02:14AM (#24339024)
You need buy-in to automation at a high enough level.

If a team manager at eg Google was complaining about how automation gets in the way and wanted to bypass it, they wouldn't last too long.

Re:Level 3 Global Outage by edoceo on Sunday August 30, 2020 @06:53PM (#24326687)
Managers who have been bitten still make this choice
Re:Level 3 Global Outage by coryrc on Sunday August 30, 2020 @07:41PM (#24326982)
Or most of what we do isn't really important so it doesn't matter if it breaks every once in a while.
Re:Level 3 Global Outage by pacificmint on Sunday August 30, 2020 @01:40PM (#24324415)
Probably not the book you are thinking off, since it’s just about the AT&T incident, but “The Day the Phones Stopped Ringing” by Leonard Lee is a detailed description of the event.

It’s been many years since I read it, but I recall it being a very interesting read.

Re:Level 3 Global Outage by tinco on Sunday August 30, 2020 @05:20PM (#24326017)
For some reason in my university almost every CS class would start with an anecdote about the Therac 25, Ariane V, and/or a couple others as a motivation on why we the class existed. It was sort of a meme.

The lessons are definitely still taught, I don't know if they're actually learned of course.. And who knows who actually taught the 737-Max software devs, I don't suppose they're fresh out of uni.

Re:Level 3 Global Outage by TheSpiceIsLife on Monday August 31, 2020 @12:59AM (#24328634)
Do management typical typically study Computer Science?
Re:Level 3 Global Outage by _heimdall on Monday August 31, 2020 @12:49PM (#24333273)
Unfortunately most people become a manager by bring a stellar independent contributor. People management and engineering are very different skills, I'm always impressed when I see someone make that jump smoothly.

I always wanted companies to hire people managers as its own career path. An engineer can be an excellent technical lead or architect, but it can feel like you started over once you're responsible for the employees, their growth, and their career path.

Re:Level 3 Global Outage by thatfunkymunki on Monday August 31, 2020 @03:43PM (#24335220)
Yeah, it just sucks that you eventually have someone making significant people management decisions without the technical knowledge of what the consequences could end up being. This would be even worse if you had people manager hiring be completely decoupled. The US military works this way and I have to say it's not the best mode.
Re:Level 3 Global Outage by tinco on Monday August 31, 2020 @03:50AM (#24329306)
Typically yes actually, the director of engineering should always be an engineer. Of course, these are hardware companies so it would probably be some kind of hardware engineer.
Re:Level 3 Global Outage by TheSpiceIsLife on Tuesday September 01, 2020 @08:58AM (#24341260)
Should.

Sure.

Re:Level 3 Global Outage by _heimdall on Monday August 31, 2020 @12:40PM (#24333164)
As a former AT&T contractor, albeit from years later, this checks out. Sat in a "red jeopardy" meeting once because a certain higher-up couldn't access the AT&T branded security system at one of his many houses.

The build that broke it was rushed out and never fully tested, adding a fairly useless feature for said higher-up that improved the UX for users with multiple houses on their account.