News for hackers. Stuff that matters.
Re:Widespread impact caused by Level 3 BGP route leak by brians on Monday November 13, 2017 @08:28AM (#15686368)
Well, there’s the problem right there. There was an individual responsible, meaning one set of hands unreviewed on a keyboard, for a change that could cause a global outage.

If I’m that person responsible, I’m going to hire two staff, ask them each to write the command scripts, justify any differences, produce a consensus script for my review, and then implement it. That seems like the minimum level of responsible engineering. Lint tools, notices of novel commands, and other ornaments have a place too, but the core idea is this:

The person with hands on keyboard is not the individual responsible for this error.

Re:Widespread impact caused by Level 3 BGP route leak by jeremyjh on Monday November 13, 2017 @08:53AM (#15686502)
Where did they say the change was not reviewed?
Re:Widespread impact caused by Level 3 BGP route leak by jdwithit on Monday November 13, 2017 @09:57AM (#15686873)
It's implied by their claim that there is a single responsible individual. If someone else reviewed it and said "yep, looks good, deploy the change" how is there one "individual responsible"?

The idea of an isolated root cause or single human error in the failure of complex systems is bogus anyway. I'm a huge fan of the work in this area championed by John Allspaw [0].

[0]: https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...

Re:Widespread impact caused by Level 3 BGP route leak by dboreham on Monday November 13, 2017 @10:14AM (#15686989)
Indeed. Someone runs the department that individual works in and allowed this kind of uncontrolled process. Someone is that person’s boss and should have been asking how well controlled are our processes, and so on. Turned Turtles all the way up..
Re:Widespread impact caused by Level 3 BGP route leak by jlgaddis on Tuesday November 14, 2017 @12:07AM (#15692831)
This could have been as simple as the individual accidentally "missing" a single line of a large, multi-line configuration when copy/pasting it into the router's console -- after all of the config review, etc., already occurred.

I'm not sure what vendor's gear was in use in this particular case, but the configs for a BGP peering session are typically (as mentioned above) large, multi-line configurations. For example, here's the (slightly redacted) configuration for one single BGP session on one of my routers:

  neighbor 10.10.10.10 remote-as 65432
  neighbor 10.10.10.10 transport connection-mode passive
  neighbor 10.10.10.10 description TO CUST FOO BAR INC ...
  neighbor 10.10.10.10 ebgp-multihop 3
  neighbor 10.10.10.10 update-source Loopback0
  neighbor 10.10.10.10 send-community
  neighbor 10.10.10.10 soft-reconfiguration inbound
  neighbor 10.10.10.10 prefix-list ACCEPTED-PREFIXES-AS65432 in
  neighbor 10.10.10.10 prefix-list ADVERTISED-PREFIXES-AS65432 out
  neighbor 10.10.10.10 password 7 0123456789ABCDEF0123456789ABCDEF
  neighbor 10.10.10.10 maximum-prefix 200
  neighbor 10.10.10.10 default-originate
That doesn't even include the associate prefix lists (or filter lists or route-maps or ...). All it would take is fat-fingering/typo'ing one of these lines or missing one to cause some very unintended effects.

Since we don't know exactly what happened, it's easy to say "they should've done this" or "they didn't do that". In reality, however, we simply don't know what they did or didn't do. You've shown no evidence that they didn't do any of the things you mention and, in some cases, you can do all of that and still have things go wrong.