News for hackers. Stuff that matters.
Re:An Update on Our Outage by wpietri on Monday November 01, 2021 @10:22AM (#29066909)
So true. And also at the systems level. In most places I've seen, the #1 priority is hitting arbitrary executive feature/date goals, not maintaining robust systems. At some point, the shit will hit the fan, causing a "Why didn't you do perfectly the thing that wasn't a real priority?!?" reaction and a temporary lurch toward robustness. Although often the lurch will be less about actual robustness and more toward performative addition of control mechanisms like more layers of review.

I remember one large company I consulted for in the mid-aughts. Before I got there, they had an outage so severe it was on CNN and caused a short but notable dip in their stock price. By the time I got there, the ops people were absolutely dominant. All changes had to go through their review board, and woe be unto any project that they raised an eyebrow at.

Of course, the real problem was that developers were scheduled out 18 months in advance to work on a list of projects they hadn't been involved in estimating and where often they hadn't seen the code before being swapped onto the project. Dates were absolutely not allowed to slip, so it came out in frantic overwork, which meant a lot of bad/confusing code. Meaning more likelihood of failure and no time to think about systemic issues and impacts.

Re:An Update on Our Outage by gsibble on Monday November 01, 2021 @10:24AM (#29066941)
I worked at a big fintech company about to go public in a SPAC worth $4 billion. They were a unicorn when I was there. Literally everything was in a GCP MySQL Database......and they refused to pay for a hot spare. They had backups but nothing for redundancy. We had downtime almost every week.
Re:An Update on Our Outage by miketery on Monday November 01, 2021 @11:32AM (#29068064)
Guessing its SoFi.
Re:An Update on Our Outage by presidentender on Monday November 01, 2021 @01:51PM (#29070274)
SoFi's devops function is extremely robust, if overcomplicated, and they've got redundancy for all their application servers and production databases. They're also an AWS shop, not GCP.
Re:An Update on Our Outage by enjoy-your-stay on Tuesday November 02, 2021 @04:38PM (#29087031)
Lol, I can just hear the politicians now.

"Tough on bugs, and tough on the causes of bugs!"

As they add even more time-sapping process to every release.