Re:An update on Sunday’s service disruption
by gundmc on Tuesday June 04, 2019 @12:52AM
(#20091999)
Code is reviewed, but I'm not aware of any companies where terminal commands are reviewed before each execution (though maybe they should be - it seems like every major cloud outage is config related).
It sounds like the change was reviewed and approved but incorrectly pushed.
[ Parent
| on HN ]
Re:An update on Sunday’s service disruption
by eyjafjallajokul on Tuesday June 04, 2019 @01:01AM
(#20092057)
We do at AWS. Not all commands though, but most commands that we audit and find are dangerous.
See a similar outage in S3 from 2 years ago - https://aws.amazon.com/message/41926/
Re:An update on Sunday’s service disruption
by thundergolfer on Tuesday June 04, 2019 @01:32AM
(#20092194)
How do this code-review of cmds work? Does the command get saved to a file, and then that file is reviewed like regular source-code, and then when it is approved the cmd is copy-pasted back to the terminal and run?
That above seems pretty clunky, so it's very likely not what happens.
Re:An update on Sunday’s service disruption
by zedgerman on Tuesday June 04, 2019 @02:24AM
(#20092428)
I’ve seen scripts get checked in and deployed just like you would a new service (code). Same Code Review process and same release pipeline.
In this particular case, commands that were run on a Production machine were by-design limited to what they can do and affect (mostly just the physical host they’re run on or a few hosts in the logical group of hosts they belong to).