Also, make sure everyone knows you are going on vacation once you resolve things.
Sure it is fun, but the stakes can be high and the stress gets piled on.
It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off!
Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed.
Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.
To be fair, the number of fires is finite, new ones are just getting started faster than they are put out.
If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.
And for some reason warehouse managers seem to swear by it, so it still gets a decent number of new customers - meaning those customers need implementers. And since it's a giant pile of hacks, the average deployment needs a whole lot of customizations - so more implementation man-hours, and a steady stream of maintenance man-hours. Thus, I get pinged every once in awhile for some long-term contract. Too bad they didn't ping me when I was actually looking for work last year, else I probably would've accepted one for the hell of it. Still tempted to; would be an interesting side job, albeit probably soul crushing.
However, not only is not everything a true positive, probably only about 0.001% of things are true positives, among a sea of alerts and reports and dashboards across myriad systems. Some coming from your SIEM, some generated by security appliances and products, some from internal employee reports.
An ideal place will have people who continuously work on trying to reduce alert fatigue and false positive noise - but, in practice, at most big companies it's probably like working at a fire station and getting hundreds of dispatch calls per hour, every hour, every day, each about a potential fire at a different residence. And then you drive up and see they just used the stove for a few minutes or a character said the word "fire" in a TV show they were watching.
But you have to urgently show up every time no matter what because, occasionally, the house actually is engulfed in flames and might be on the verge of igniting the whole town.
It is... interesting. I have been doing this over last 4 years. The biggest problem, as I pointed out before, is the disconnect between what companies claim they would do to fix the problems vs. what they would actually do. So ~90% of selling is repeatedly explaining the same things to different people whose jobs are in the eminent danger of being restructured/deprioritized/eliminated if your services are successfully engaged.
<multiple layers down, Jack, head of IT in charge of backups>: I'm uncomfortable sharing this information with the outside party. I will need to get the approvals for <blah blah blah blah>
...
CEO: Yes, tell them to do this.
...
Jack, head of IT in charge of backups: Ok, you can have this access.
Consultant: Great. <comes back ten minutes later> It says I'm not authorized to perform this operation. It is a blocker, could this be fixed so I could continue?
Jack: This was not in the scope of authorization that I have received. I'm uncomfortable providing this level of access without additional authorization.
<repeat>
The guy did very creative reporting though :-)
And the problem was not fixed, of course.
I tried to help him as much as I could, but at some point, the thing is so intricate that you basically give up.
When you’re in this role your goal and job is to build relationships and help people. That includes Jack. The job is getting done with or without Jacks help. So he can either jump onboard or he can be removed and someone else can be added to give access.
In the end people like Jack benefit nothing from being gatekeepers. Because the third time I have to ask for permissions to be changed - I’m asking for Jack to be removed from the task and someone else to be put in place.
When you are in a firefighting role, your job is to fix the fire, not appease Jack and build relationship with him, the person whose practices created the situation to begin with. Unfortunately, even in those cases the highest level of stakeholder who engaged your firefighting service may not be willing to tell Jack's manager or Jack that he will do what he is told or he will be removed.
Jack's gatekeeping is what keeps Jack employed. The company's ultimate bosses will need to make decision if they want to remove Jack and solve a problem or if they do not care that much that the problem is quickly solved. In the vast majority of cases no one in the company who can fire Jack wants to be seen as a bad guy.
If you are an external consultant (with no prior knowledge of the company's systems/processes/hierarchies) brought in to "fight" a particular fire, how do quickly get ramped up to a level of knowledge where you can analyze the root cause of/recommend a fix for a particular "fire"?
"If you do things well enough, nobody will think you did anything at all."
In war room after war room there has always been at least one person who would be categorized as 'disorganized' by their coworkers. When thrown into a stressful situation they either don't change or become more focused, while the 'organized' people crumple like tissue paper, so Mr Disorganized ends up either fixing the problem, keeping everyone on task until it does get fixed, or insisted on some process/tool improvement that made the problem easier to identify or fix.
I've been diagnosed with ADHD and I'm not sure I'd qualify as having that trait. So I don't believe this a general sweeping statement you can make for everyone with that, or if your boss even had ADHD.
But maybe check this out and ask if it rings any bells: https://www.nhs.uk/conditions/attention-deficit-hyperactivit...
I have ADHD and recently started an SRE squad in our company since I tend to be the go-to person to fix any production issues.
I had never really thought of a connection between those two things until now.
While you can and should be the cool head in the room, it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization and management from the consequences of its own failures. By all means do your best fix the immediate problem but get the hell out as soon as you can.
The same dysfunction that led to disaster after disaster is likely to lead to very little reward (if any) for actually being the hero, also.
>We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.
It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost.
I often wonder what proportion of backed up files and systems are actually in a state to be restored and have the necessary tools and expertise to do it in a timely manner.
An even worse problem in some places, including the one I worked at, is a tendency for users to assume that backups are kept forever and for the management to totally neglect archiving of important documents such as specifications, drawings, and design calculations of products that have an expected lifetime of over fifty years.
I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a year and taking a set out of rotation and into archive.
(GDPR and earlier laws, etc.)
The best way to delete user data is to encrypt it all with per-user keys so they can be shredded immediately upon request. Backups of user keys themselves can be very short-lived since the keys are static and the lifetime is scoped to GDPR/other laws. Then the encrypted data can be archived indefinitely, so long as encryption keys are never stored in those archives.
Up until now, they didn't know what 'backup not working' meant. Now they know.
Not being bitter -- there was a failure of communication from both parts.
Typically, this kind of decisions are made more or less implicitly during long meetings about priorities, often without any minutes. Managers will presumably try to hold (senior) engineers accountable as they had not made clear enough how important that was.
I obviously don't know who is right here.
Thinking you're fucked is unhelpful, because you start talking about resumes and expecting some deus ex machina to appear and resolve the situation (for good or ill).