The only way I could get to solving the problem was by drawing a line in the sand and refusing to work on symptoms anymore. And the only reason that was effective was because my team was so understaffed that my refusal effectively brought all feature work to a halt.
Even then, “solve the problem” buy in was shaky and constantly dismissed/side-stepped. It took a full end to end proof of concept exploit against a production system, being incredibly careful not to actually cause harm, to get support from the top (which I conjured in less than an hour and regret holding those cards as long as I did, but I was seriously worried about how I could demonstrate the vulnerability without exploiting it harmfully).
If you haven’t been shot down by an expendable in the past, the best course of action is to find the most complementary position historically taken by leadership, twist to your objective and then credit them publicly with the vision.
The gap can be reconciled privately later.
You missed that part. "Here is the email chain it was discussed in, including the bit where I raised the concern of what could happen" is generally pretty solid CYA.
What has your experience been? Have you had success reprioritizing necessary and critical fixes that were previously not in scope postmortem?
And in both situations even when said thing exploded in a bloody mess I struggled to get the buy in to give the fix the full attention it needed because "we're losing money not having these features" (I mean no, you're not, but ok)
Hence my incredulity but honest curiosity, being an ops guy who over-prepares and tries to keep his leaders as informed and prepared to make a decision as possible, but still feels like he's often left holding the bag is dang exhausting, you know?
Yes, we knew about it. Also, here's all these other things we know about that will result in similar catastrophes
Maybe I'm overthinking it (this happens sometimes, I'm working on it), but waiting around for a security crisis to happen when a solution is in sight, even when facing difficulty prioritizing the work seems dangerous to one's career.
But I admit not knowing a lot of things most engineers just assume are a given, lately.
In a push shop (e.g. most non-tech businesses / legacy tech), one is forbidden from working on a thing without management signing off on it and pushing the task to you.
In a pull shop, management communicates priorities (increase reliability) and engineering teams pull tasks to fulfill that.
Believe parent is talking about the former, which is always a cluster&#+@ of technically clueless middle management.
That's been a real struggle lately, and I want to get better at this to avoid these kinds of push shops you're describing
On a side note, I've found push shops tend to be highly correlated with length of position (i.e. "What's the longest someone has ever been on this specific team without transferring or pursuing new opportunities?"). But that's probably more accurately identifying heavily regulated industries, which tend to be push shops do to the Byzantine sign-offs required.
I would say don't ask, "Tell me about a time your team originated an idea, took it to production, and the net impact." Because even push shops have the odd exception that can be rattled off. It's more talking about the typical workflow.
The best I could probably come up with is "Take me through a normal task for your team, from idea origination to prod deployment."
If they start with "We were told to do X", then "By whom? And what did they tell you?" (To the latter, did they specify exactly how or just the end goal?)
Everything about that situation is dangerous to our careers: saying something, saying nothing, saying a lot, stomping our feet, saying we told them so.
But don’t second guess yourself. You’re expressing exactly the right attitude toward addressing things before they’re an emergency. The people who are using those emergencies to get things done agree, we’re just accustomed to getting shut down unless there’s no other option.
The problem is really of incentives and risk aversion. When there's a tough decision to make without enough clarity on the outcome, people will always prefer the status quo.
Since the future and the result of our actions is uncertain, society is basically executing a search algorithm. This covid crisis has focussed society on finding a local optimum. Though this focussed search will only get us so far. To find a better _global_ optimum, we need a lot of random, seemingly counterproductive search.
Don't get me wrong, I strongly dislike a lot about the current state of society. But I think it's irrational to think that we can predict what will be useful in a hundred years.
Keep in mind that if the pandemic results in extermination of the common cold, in the alternative world without a pandemic we would most likely also achieve it, just slower. It isn't a matter of now or never.
The situation is different when COVID is a catalyst because we are already spending $19 billion (for example) on a COVID cure, and along the way we have learned that if we spend another $1 billion there's a 90% chance we'll cure the common cold.
Having been forced into this situation, it makes sense to go the last mile. That doesn't mean the whole trip would have been clearly worthwhile from the start.
On the other hand, it is an indictment when that's a concern, when a private company trivially spends comparable amounts for buying things like Instagram and WhatsApp to push more ads, milk more personal details, and poison more minds with BS and waste more personal time...
Covid is a specific class of virus (coronavirus) of which there are relatively few variants.
If it cost $20 Billion to deal with Covid, you'd have to spend at least 100 times that for each and every individual strain of virus that tends to be viewed as one 'common cold.'
Oh, and we've already been trying to cure the 'common cold' for decades. The problem is that there's just too many of 'em.