Re:How we spent $30k in Firebase in less than 72 hours
by michalprzy on Thursday August 02, 2018 @02:36AM
(#17669631)
Unfortunately I often see developers asking for complex diagnose in urgent emergencies. When it's literally like a 30 minutes reaction time frame.
So from business pov there should be first quick and dirty unperfect analysis which is necessary for emergency decisions.
Sometimes you can just prolong caching time as quick fix, but sometimes there is no such option.
Programmers like nice and calm working environment but business often is very different place. So, please have in mind what business goals are.
And yes, I see there is a lot of stupid managers in It and basically you're right, we should use precise data. Just one point: when available...
[ Parent
| on HN ]
Re:How we spent $30k in Firebase in less than 72 hours
by jon-wood on Thursday August 02, 2018 @04:08AM
(#17669956)
The first thing I'm likely to say to my team when an outage occurs is something along the lines of "stop the bleeding". That might mean bypassing the affected service if possible, rolling back a recent release, or reducing the amount of traffic (we're fortunate enough to have most of our traffic coming from sources we can throw a kill switch on).
However we go about it, the first priority is to give ourselves some space to properly analyse the issue and find the real solution without the rest of the business worrying loudly about things being broken.