Re:Summary of the Amazon S3 Service Disruption
by gigatexal on Thursday March 02, 2017 @01:01PM
(#13776465)
Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.
[ Parent
| on HN ]
Re:Summary of the Amazon S3 Service Disruption
by duskwuff on Thursday March 02, 2017 @01:37PM
(#13776839)
FWIW: Setting "safe-updates=1" in ~/.my.cnf will require UPDATE and DELETE statements in the client to have a WHERE clause which references a key. It's not perfect protection, but it will save you from a lot of mistakes.
Re:Summary of the Amazon S3 Service Disruption
by amichal on Thursday March 02, 2017 @04:14PM
(#13778281)
That's awesome,
My worst DELETE fail however was:
DELETE * FROM table WHERE [long condition that resolves to true for all records]
Now i write
SELECT or SELECT COUNT(*) over and over again until i see the data i expect and then change it to a DELETE/UPDATE.It's not my personal habit but some folks I know turn off auto commit and BEGIN a transaction every time they enter an interactive SQL sessions. They then default to ROLLBACK at least once before COMMITing them.
That and having a user with read-only permissions or a read replica
Re:Summary of the Amazon S3 Service Disruption
by tacon on Thursday March 02, 2017 @04:31PM
(#13778432)
Is there a collection of data safety tips like this somewhere? I never knew this existed. What else am I missing?
Re:Summary of the Amazon S3 Service Disruption
by gigatexal on Thursday March 02, 2017 @01:39PM
(#13776854)
hmm, that's kinda cool. I'm in a MS shop and I don't know if SMSS has the same feature. My manager just looked at me and said "welp, go restore the table and be more careful next time." I was a new DBA at the time, still, kinda new.
Re:Summary of the Amazon S3 Service Disruption
by jsudhams on Thursday March 02, 2017 @10:26PM
(#13780523)
USE BEGIN TRANS as mentioned above
Re:Summary of the Amazon S3 Service Disruption
by vocatus_gate on Thursday March 02, 2017 @02:28PM
(#13777353)
I once brought down our entire production XenServer cluster group by issuing a "shutdown now" in the wrong SSH window. Needless to say it was a bad feeling watching Nagios go crazy and realizing what had just happened.
Re:Summary of the Amazon S3 Service Disruption
by Piskvorrr on Thursday March 02, 2017 @05:14PM
(#13778773)
root@baz # shutdown now
W: molly-guard: SSH session detected!
Please type in hostname of the machine to shutdown: foo
Good thing I asked; I won't shutdown baz ...
Surprising to see such a simple protection neglected.
Re:Summary of the Amazon S3 Service Disruption
by twunde on Thursday March 02, 2017 @07:15PM
(#13779654)
I don't know how well-known molly-guard is, but I've never heard of it before. Definitely enabling it on my servers next week.
Re:Summary of the Amazon S3 Service Disruption
by Piskvorrr on Friday March 03, 2017 @02:58PM
(#13785795)
Interesting. Up until now I've considered it well-known to the point of ubiquity :)
Re:Summary of the Amazon S3 Service Disruption
by gigatexal on Thursday March 02, 2017 @03:22PM
(#13777812)
Oh crap! Yeah I bet you were pretty panicked. My update statement destroyed data that my team used all the time so I was worried I'd get fired. Luckily that wasn't the case.
Re:Summary of the Amazon S3 Service Disruption
by saalweachter on Thursday March 02, 2017 @02:14PM
(#13777227)
Fat fingers are just nature's way of making sure you test your back-up & restore procedures periodically :-)
Re:Summary of the Amazon S3 Service Disruption
by wakkaflokka on Thursday March 02, 2017 @08:34PM
(#13780071)
When I first started using Linux and wanted to do some housecleaning, I did "rm -r (asterix) in a folder. Cleaned up everything, no prob. Then went to some more folders, hit the up arrow on my keyboard fast to get to a command I had used before. Hit 'enter' before my brain realized I had landed on rm -r (asterix) and not the right command. Never used that command again.