Back when we had servers in an onsite DC we lost a raid card and the system I was developing went down. We had the fancy support so a tech was out with the card replaced in a couple hours, then we had to restore from tape backup. All in all, a non-critical system was down for most of a business day. My bosses boss stormed in, upset he couldn't pull a report, and asked how do we prevent this in the future. I responded at a minimum we had to double the cost for a hot standby, and he said 'never mind' and walked out.
Re:AWS us-east-1 down
by d8dUf7KjBEYCk7Q on Tuesday June 13, 2023 @04:45PM
(#36316951)
That sounds nice. My boss's boss is usually the one storming in, and he usually says "okay let's do it", and then I have to implement it in a week...
That’s why you always BofH the estimates to include some fun toys for yourself, too
Always be prepared for alien invasion
Just need to deploy your service on Mars AND Earth. Duh
And you thought the current time zone confusion was bad. Now you have two sets of time zones, and a varying delay of about 5 to 21 minutes between them. Oh, the joy!
note to self: synchronous replication may be a problem
Re:AWS us-east-1 down
by elzbardico on Wednesday June 14, 2023 @02:01PM
(#36329904)
Multi-planetary-AZ DB cluster deployments. Putting the emphasis on "Eventual" in eventual consistency.
Go for a walk before retrying reading from this replica!
"Briefly describe the '$$$$$$$' through '$$$$$$$$$$$$$' situations. Can't leave money lying on the table."
- memo from Enterprise Sales Dept.
Alien invasion resistant is spy novel / agents of shield level of multiple, redundant, isolated, off the normal books, safehouses + bases.
Short of alien invasion level are strategic military resistance levels to global/regional wars with differing levels of weapons and devastation.
Re:AWS us-east-1 down
by underbluewaters on Tuesday June 13, 2023 @05:26PM
(#36317640)
This should be logarithmic
then you get to eat popcorn when stuff explodes.