Ask HN: Best “I brought down production” story?

Re:Ask HN: Best “I brought down production” story? by alfiedotwtf on Saturday June 26, 2021 @07:11PM (#27646864)

i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.

[ Parent | on HN ]

Re:Ask HN: Best “I brought down production” story? by chasd00 on Saturday June 26, 2021 @11:29PM (#27648311)

I had a DBA call me on my way back from lunch absolutely sobbing after fat fingering a semicolon before the where clause while logged in to Prod as root.

we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.

[ Parent | on HN ]

Re:Ask HN: Best “I brought down production” story? by rrauenza on Sunday June 27, 2021 @12:24AM (#27648570)

I was taught to also use a transaction and then check how many rows were affected before committing.

[ Parent | on HN ]

Re:Ask HN: Best “I brought down production” story? by sodality2 on Sunday June 27, 2021 @08:51AM (#27651183)

As was Tom Scott: https://youtu.be/X6NJkWbM1xk

[ Parent | on HN ]

Re:Ask HN: Best “I brought down production” story? by rrauenza on Tuesday June 29, 2021 @03:26PM (#27680594)

I wonder if archive.org was around enough back then to capture any of it.

[ Parent | on HN ]

Re:Ask HN: Best “I brought down production” story? by chasd00 on Sunday June 27, 2021 @09:17AM (#27651335)

Many mistakes were made prior in order to get her in the situation to begin with. The semicolon was just the icing on the cake.

[ Parent | on HN ]

Re:Ask HN: Best “I brought down production” story? by kryptk on Saturday June 26, 2021 @10:19PM (#27647939)

When your database is too big to backup that's a lot like a bank being too big to fail. Unwise long term strategy.

[ Parent | on HN ]

Re:Ask HN: Best “I brought down production” story? by alfiedotwtf on Sunday June 27, 2021 @12:26AM (#27648581)

Terabytes across 25+ databases not co-located and all being masters replicating to each other, in 2009. Not sure how you would have done it.

[ Parent | on HN ]

Re:Ask HN: Best “I brought down production” story? by mianas on Sunday June 27, 2021 @06:57PM (#27656241)

I got myself into the habit of opening transactions before doing anything that could alter data by hand. Huge panic-saver.

Even earlier, I got into the habit of typing the where clause before typing the from clause. Not much of a change, but it feels more fail-safe.

[ Parent | on HN ]