Re:Ask HN: Best “I brought down production” story?
by onion2k on Saturday June 26, 2021 @06:02PM
(#27646337)
And this is why we use things like database users that don't have delete permission, and row-level security so users can delete things that don't belong to them.
I have learned this from a very similar experience.
Re:Ask HN: Best “I brought down production” story?
by bombcar on Saturday June 26, 2021 @11:27PM
(#27648298)
And why it’s often best to mark a record deleted and then have a reaper remove the records at a later point. But you must make sure all normal queries don’t see deleted items.
Re:Ask HN: Best “I brought down production” story?
by truetraveller on Saturday June 26, 2021 @11:49PM
(#27648409)
That's over engineering. At this point, just rely on PITR. FWIW, postgres does have a "reaper" via vacuum, but not for the purpose of safety. but rather to allow for mvcc.
Re:Ask HN: Best “I brought down production” story?
by brutal_chaos_ on Sunday June 27, 2021 @02:33AM
(#27649161)
Hi, if I may, what does PITR stand for?
Re:Ask HN: Best “I brought down production” story?
by brutal_chaos_ on Sunday June 27, 2021 @04:11AM
(#27649613)
Thanks!
Re:Ask HN: Best “I brought down production” story?
by yyyk on Sunday June 27, 2021 @10:26AM
(#27651817)
There are other advantages for soft-deletes, like not having to worry about FKs.
Re:Ask HN: Best “I brought down production” story?
by throwaway6734 on Saturday June 26, 2021 @05:29PM
(#27646056)
>One night in bed I realized that if someone hit submit on the delete screen without filling in any criteria it would just delete the whole database.
how
Re:Ask HN: Best “I brought down production” story?
by breakfastduck on Saturday June 26, 2021 @05:33PM
(#27646097)
Lack of validation?
I.e if no criteria, it could be sending a DELETE message with no where clause in SQL land.
Re:Ask HN: Best “I brought down production” story?
by alfiedotwtf on Saturday June 26, 2021 @07:11PM
(#27646864)
i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.
Re:Ask HN: Best “I brought down production” story?
by chasd00 on Saturday June 26, 2021 @11:29PM
(#27648311)
I had a DBA call me on my way back from lunch absolutely sobbing after fat fingering a semicolon before the where clause while logged in to Prod as root.
we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.
Re:Ask HN: Best “I brought down production” story?
by rrauenza on Sunday June 27, 2021 @12:24AM
(#27648570)
I was taught to also use a transaction and then check how many rows were affected before committing.
Re:Ask HN: Best “I brought down production” story?
by sodality2 on Sunday June 27, 2021 @08:51AM
(#27651183)
As was Tom Scott: https://youtu.be/X6NJkWbM1xk
Re:Ask HN: Best “I brought down production” story?
by rrauenza on Tuesday June 29, 2021 @03:26PM
(#27680594)
I wonder if archive.org was around enough back then to capture any of it.
Re:Ask HN: Best “I brought down production” story?
by chasd00 on Sunday June 27, 2021 @09:17AM
(#27651335)
Many mistakes were made prior in order to get her in the situation to begin with. The semicolon was just the icing on the cake.
Re:Ask HN: Best “I brought down production” story?
by kryptk on Saturday June 26, 2021 @10:19PM
(#27647939)
When your database is too big to backup that's a lot like a bank being too big to fail. Unwise long term strategy.
Re:Ask HN: Best “I brought down production” story?
by alfiedotwtf on Sunday June 27, 2021 @12:26AM
(#27648581)
Terabytes across 25+ databases not co-located and all being masters replicating to each other, in 2009. Not sure how you would have done it.
Re:Ask HN: Best “I brought down production” story?
by mianas on Sunday June 27, 2021 @06:57PM
(#27656241)
I got myself into the habit of opening transactions before doing anything that could alter data by hand.
Huge panic-saver.
Even earlier, I got into the habit of typing the where clause before typing the from clause. Not much of a change, but it feels more fail-safe.
Re:Ask HN: Best “I brought down production” story?
by r00f on Sunday June 27, 2021 @03:49PM
(#27654659)
in one web shop i worked with, developers who were fanaticslly using constructions "WHERE user LIKE '%$user%' "
we've had some really heated debates, but nobody would listen to me. and then one day DELETE happened where $user was empty. they spent half day restoring the database, but nobody made any conclusions and nothing changed...
Re:Ask HN: Best “I brought down production” story?
by e67f70028a46fba on Saturday June 26, 2021 @05:33PM
(#27646098)
Easy: no form input no condition on the delete
aaand it’s gone
One night in bed I realized that if someone hit submit on the delete screen without filling in any criteria it would just delete the whole database.
Not a fun drive in.
Yes, we drove in in those days.