News for hackers. Stuff that matters.
Re:Google outage – resolved by ignoramous on Monday December 14, 2020 @08:00AM (#25416956)
Same.

At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

Re:Google outage – resolved by papito on Monday December 14, 2020 @08:18AM (#25417130)
Jesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.
Re:Google outage – resolved by ignoramous on Monday December 14, 2020 @08:36AM (#25417302)
Yes but that was eons ago. The safeguards are well and truly in-place now. Not just one, several in fact.
Re:Google outage – resolved by qz2 on Monday December 14, 2020 @08:51AM (#25417465)
Apart from the ones that they haven't worked out yet :)
Re:Google outage – resolved by zentiggr on Monday December 14, 2020 @09:30AM (#25417926)
When I meet the engineer who can design for the unknown unknowns, I will bow to them.
Re:Google outage – resolved by papito on Monday December 14, 2020 @09:41AM (#25418055)
The trick is to be paranoid. You literally sit down and think exclusively about what COULD go wrong.
Re:Google outage – resolved by GreenWatermelon on Monday December 14, 2020 @04:57PM (#25423935)
Anxiety is a bitch.
Re:Google outage – resolved by sitkack on Tuesday December 15, 2020 @01:11AM (#25427521)
Formal methods for your formal methods. And never shipping on Friday.
Re:Google outage – resolved by marcus_holmes on Monday December 14, 2020 @09:24AM (#25417840)
colorblind (red/green) person here - 5% of the male population just don't see color enough for it to be an important visual clue.

So sure, color-code your environments, but if you find someone about to do something to a red environment that they clearly should only be doing to a green environment, just check if they're seeing what you're seeing before you sack them ;)

Re:Google outage – resolved by warrenm on Monday December 21, 2020 @11:24AM (#25496326)
My primary customer right now color codes both with "actual color", and with words - ie the RED environment is tagged with red color bars, and also big [black] letters in the [red] bars reading "RED"
Re:Google outage – resolved by ardy42 on Monday December 14, 2020 @09:56AM (#25418234)
> At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

It seems like a design flaw for actions like that to be so easy. E.g.

> Hey, we detected you want to delete an AWS region. Please have an authorized coworker enter their credentials to second your command.

Re:Google outage – resolved by ignoramous on Tuesday December 15, 2020 @10:40AM (#25431235)
If it was indeed in-production, I'd never in a million years have had access rights to delete the stack. Those are gated just the way one would imagine they should be.

The service stack for the region (and not an entire region itself) looked like prod, but wasn't. It made me feel like shit anyway.

Re:Google outage – resolved by znpy on Monday December 14, 2020 @09:45AM (#25418103)
It reminds me of this: https://www.youtube.com/watch?v=30jNsCVLpAE -- "GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill"
Re:Google outage – resolved by ayoubElk on Monday December 14, 2020 @09:12AM (#25417711)
Irrelevant to the discussion, but I just wanted to say thank you for the categorized list of users I can follow on your profile!
Re:Google outage – resolved by laddng on Monday December 14, 2020 @10:24AM (#25418557)
Which tool do you use to follow users on HN?
Re:Google outage – resolved by ignoramous on Monday December 14, 2020 @11:01AM (#25419034)
There used to be hystry.com [0] but it isn't functional anymore.

Another workflow, though cumbersome, is: Search for a username on hn.algolia, select "comments" and "past months" as filters, then press enter.

Ex: https://hn.algolia.com/?dateRange=pastMonth&query=nostrademo...

[0] https://news.ycombinator.com/item?id=71827

Re:Google outage – resolved by laddng on Monday December 14, 2020 @11:35AM (#25419506)
Thanks for sharing those tips!
Re:Google outage – resolved by rptr on Monday December 14, 2020 @09:26AM (#25417871)
wow, that list is crazy. thanks OP.
Re:Google outage – resolved by robertlagrant on Monday December 14, 2020 @08:07AM (#25417025)
Who automates the automators? :)
Re:Google outage – resolved by ahachete on Monday December 14, 2020 @08:56AM (#25417526)
Doesn't AWS (and every big cloud/enterprise) follow best-practices for production operation like FIT-ACER? https://pythian.com/uncategorized/fit-acer-dba-checklist/

That's even more surprising to me.