News for hackers. Stuff that matters.

A comment on Ask HN: Best “I brought down production” story?.

Re:Ask HN: Best “I brought down production” story? by cvrjk on Saturday June 26, 2021 @11:29PM (#27648306)
Sent in a SQL command deleting multiple million records (intended) wrapped in a single transaction (not intended). The replication queues could not keep up and failed, bringing down most of the replicas. Master server kept trying to recover and maxed out all connections - no DBA could log in to perform manual recovery. We had to hard reboot not knowing what state the system is in and how long it'll take to fully recover. Did I mention that this was a few hours before trading was going to begin?

TBH, my team was very gracious about it and the RCA focused purely on the events that occurred and how to never let if happen it again. No blame game at all.

[ Parent | on HN ]
Re:Ask HN: Best “I brought down production” story? by monkeydreams on Saturday June 26, 2021 @11:39PM (#27648365)
> TBH, my team was very gracious about it and the RCA focused purely on the events that occurred and how to never let if happen it again. No blame game at all.

Which is how a PIR, PER or PCR should be. If you don't understand why someone makes a mistake, you can't avoid future mistakes.

[ Parent | on HN ]
Re:Ask HN: Best “I brought down production” story? by brutal_chaos_ on Sunday June 27, 2021 @02:31AM (#27649152)
I understand SQL, DBA and TBH, but what do RCA, PIR, PER, and PCR stand for?
[ Parent | on HN ]
Re:Ask HN: Best “I brought down production” story? by robmobz on Sunday June 27, 2021 @03:26AM (#27649407)
RCA is "Root Cause Analysis" and I assume PIR is "Post Incident Review". I don't know PER or PCR.
[ Parent | on HN ]
Re:Ask HN: Best “I brought down production” story? by monkeydreams on Sunday June 27, 2021 @06:47PM (#27656192)
"Post Event Review" and "Post Change Review".
[ Parent | on HN ]
Re:Ask HN: Best “I brought down production” story? by exikyut on Sunday June 27, 2021 @01:09AM (#27648783)
Hmmm. I can't help but wonder if maybe the database engine should have caught the queue overflow (even if the event occurred over the network) and failed the transaction.
[ Parent | on HN ]