Re:Ask HN: Best “I brought down production” story?
by breakfastduck on Saturday June 26, 2021 @05:33PM
(#27646097)
Lack of validation?
I.e if no criteria, it could be sending a DELETE message with no where clause in SQL land.
Re:Ask HN: Best “I brought down production” story?
by alfiedotwtf on Saturday June 26, 2021 @07:11PM
(#27646864)
i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.
Re:Ask HN: Best “I brought down production” story?
by chasd00 on Saturday June 26, 2021 @11:29PM
(#27648311)
I had a DBA call me on my way back from lunch absolutely sobbing after fat fingering a semicolon before the where clause while logged in to Prod as root.
we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.
Re:Ask HN: Best “I brought down production” story?
by rrauenza on Sunday June 27, 2021 @12:24AM
(#27648570)
I was taught to also use a transaction and then check how many rows were affected before committing.
Re:Ask HN: Best “I brought down production” story?
by sodality2 on Sunday June 27, 2021 @08:51AM
(#27651183)
As was Tom Scott: https://youtu.be/X6NJkWbM1xk
Re:Ask HN: Best “I brought down production” story?
by rrauenza on Tuesday June 29, 2021 @03:26PM
(#27680594)
I wonder if archive.org was around enough back then to capture any of it.
Re:Ask HN: Best “I brought down production” story?
by chasd00 on Sunday June 27, 2021 @09:17AM
(#27651335)
Many mistakes were made prior in order to get her in the situation to begin with. The semicolon was just the icing on the cake.
Re:Ask HN: Best “I brought down production” story?
by kryptk on Saturday June 26, 2021 @10:19PM
(#27647939)
When your database is too big to backup that's a lot like a bank being too big to fail. Unwise long term strategy.
Re:Ask HN: Best “I brought down production” story?
by alfiedotwtf on Sunday June 27, 2021 @12:26AM
(#27648581)
Terabytes across 25+ databases not co-located and all being masters replicating to each other, in 2009. Not sure how you would have done it.
Re:Ask HN: Best “I brought down production” story?
by mianas on Sunday June 27, 2021 @06:57PM
(#27656241)
I got myself into the habit of opening transactions before doing anything that could alter data by hand.
Huge panic-saver.
Even earlier, I got into the habit of typing the where clause before typing the from clause. Not much of a change, but it feels more fail-safe.
Re:Ask HN: Best “I brought down production” story?
by r00f on Sunday June 27, 2021 @03:49PM
(#27654659)
in one web shop i worked with, developers who were fanaticslly using constructions "WHERE user LIKE '%$user%' "
we've had some really heated debates, but nobody would listen to me. and then one day DELETE happened where $user was empty. they spent half day restoring the database, but nobody made any conclusions and nothing changed...
Re:Ask HN: Best “I brought down production” story?
by e67f70028a46fba on Saturday June 26, 2021 @05:33PM
(#27646098)
Easy: no form input no condition on the delete
aaand it’s gone
how