This sounds like a disorganized workplace to me. The standard thought experiment is: what happens if <supposedly important person> gets hit by a bus and dies? What if they're just in a coma for a few weeks? The response to the first satisfies a sudden "I'm quitting" and the response to the second satisfies a sudden "I'm going on vacation". Business is more forgiving of course, since both quitting and vacation planning typically have a time delay.

The simplest response for both is to delegate: "<person y> now has temporary or permanent authority in the decisions <person x> used to have". That means you need to have a person y for each person x and there needs to be enough trust that person y will make decisions that x would approve of (even if sometimes different). If you're a single-person startup, there is no person y, but then these questions aren't really relevant. As the Pinboard FAQ says, the answer to what happens is the bus will be fine.

I am going to hazard a guess: you have not been exposed to some of the nuttier regulations within this space.

The regulatory demands in financial technology, and especially in those industries where both the regulators as well as majority of the players may not be technically literate, are effectively the anti-thesis of robust and reliable engineering. When the industry regulations demand that for every potentially disruptive action (recovering from an outage is very much potentially disruptive) there has to be a named individual - not even a role, but an individual - to sign off on every single release step... This is why an outage can last for hours. An engineer, or even a lead, is with high probability not allowed to sign off on an out-of-hours deployment. The fix might take 20 minutes to identify, 5 minutes to implement, 15 minutes to test, and 10 minutes to roll out. But until the organisation can get the named [supposedly responsible] person online, the fix must wait, unreleased.

And the irony? The wider dysfunction may be imposed by the regulations, but it has been requested by the industry at large. It is the result of the industry players, majority of whom are incapable (read: technologically illiterate) of interpreting prescriptive regulations. So instead of figuring out how to meet expectations and adapt their processes, they have been lobbying for the regulators to come up with highly descriptive playbooks and step-by-step instructions. These sequences then specify rules and requirements that allow technologically incompetent players to comply, even if they do not understand why they are doing the things.

And then these regulatory playbooks become the One True Way[tm], effectively forcing dysfunctional practices, but also actively preventing any process improvements.

It's cargo-culting taken to the extreme: "do this and you will not be found to be non-compliant, even if the quality is utter shite". The worst part of this all is that the descriptive regulations make every effort to strip engineers of their agency or decision power. Every "modern" best practice is explicitly ruled as non-compliant and hence effectively illegal. I use scare quotes around modern to highlight that these practices are by no means modern and have been known since ancient times. There are research papers from 1950's that explicitly endorse [rapid] iterative bottom-up approach as the only sensible way forward.

The finance industry regulations explicitly reject them because these studies assume that engineers can be trusted to know what they are doing.

So... to anyone wondering why finance industry jobs are generally thought of as soul-sucking: that's why. They are subject to regulations, expressly designed to deprive individuals of their agency.

I have consulted in very large financial institutions. Yes, there are a lot of rules. At the same time, those rules will be 'creatively interpreted' if not skirted all the time to keep disruptions to a minimum. 'The rulebook' treatment was in fact a sure sign of internal political power games being played out.

Thanks for the great comment. I don't think we're in disagreement. Government-mandated disorganized workplaces are still disorganized... I wonder about the thought experiment for such an environment though. Tragedy has a cruel way of striking from multiple vectors. You might have a critical issue that demands immediate deployment of a fix, but just before then you learned that the only named person who can legally sign-off on the deployment was involved in a fire and is dead/missing. I suspect the decision would get escalated to someone who might not have the legal authority that the dead person had, but who has enough business authority to make the call on whether the financial damage to the company from not rolling out the fix now will exceed the fine the government will impose for not following the regulation + any potential negligence lawsuit costs if the not-fully-approved fix causes even more issues.

You're right I haven't been directly exposed to such regulations (our emergency releases also require sign-off from management but there are multiple people who can approve and different people for different stages) and I don't ever intend to be. Even for something much lighter. At my current job I refused a request from my boss/boss's boss to go through the process of obtaining a Public Trust Position which would be necessary if I needed to debug my team's product for one of our government customers. Fortunately I didn't have to quit over it, and I still got promoted later. Other people on other teams just went along with it. So I guess I'd amend my thinking (aided by your comments on lots of the regulation in finance being self-requested) that for people who get themselves into a position where it's "not possible" to go 100% on vacation, regardless of the level of dysfunction at the org, many of them generally don't mind it. If they did, they could switch jobs, and anyone concerned about accepting such a deal should weigh that instead of imagining that there's nothing to be done.

Everyone who has not worked in the heavily regulated finance industry should read the above post, because this person speaks the truth.

As someone who has never worked in the finance industry, I feel like one shouldn't lose sight of the fact that the point of it all is to not have millions of dollars just vanish at random as often happens with Bitcoin. Making software engineers happy is not the point.

I agree with your sentiment. However, I believe you are missing the forest from the trees.

The desired result of the regulations is, first and foremost, accountability. This includes, but is not limited to, audit trails and provenance. But because the regulations have been written to specify exactly One True Way[tm] of reaching an adequate level of accountability, they end up causing nasty second-order effects.

As sibling comment by PeterStuer mentions, the unfortunate end result is inventive interpretation to avoid the worst of the disruptions. So from my point of view, the rules put in place to _enforce_ accountability have created perverse incentives to _reduce_ proper accountability.

That can not be a healthy situation either in short or long term.

> There are research papers from 1950's that explicitly endorse [rapid] iterative bottom-up approach as the only sensible way forward.

Interested in this... any links?

In my sibling response I mentioned that just 5 articles have been truly valuable. In no particular order:

DOI: 10.1109/MC.2003.1204375 ; "Iterative and Incremental Development: A Brief History"

DOI: 10.1109/AGILE.2013.17 ; "Continuous Delivery? Easy! Just Change Everything (Well, Maybe It Is Not That Easy)"

DOI (with link): https://doi.org/10.1002/spip.344 ; "The agile professional culture: A source of agile quality" [the article itself is pretty light on useful material but it contains some of the strongest individual arguments I've seen, all in one package]

Direct PDF link: https://www.futureworksconsulting.com/perch/resources/pdfs/t... [the valuable material is on the first page]

DOI: 10.1.1.428.5843 ; "Evaluation of Quality AssuranceFactorsinAgile Methodologies"

---

There are plenty of other reports and papers on the topics, but they bring very little to the table on their own. These five have provided me with a decent starting point. The references to findings and reports from 1950's are in the first one, btw.

Thank you!

I'll dig out the archive when back at the office, next week. There is plenty of research into iterative/agile development, slightly less into quality, and surprisingly little on accountability.

The most valuable sources have been just 5 (yes, five) research papers / reports. Luckily they sport citations, but sadly I haven't poured over more than a fraction of them so far.

Here is original statement I reacted to: "once you're senior enough, you are never 100% on vacation. If enough goes wrong, you will get called back to work."

If fecal matter hits the rotating turbines often that you are never 100% on vacation, then you have fecal hitting around all the time. It is not like seniors would be on vacation 50% time of year.

> Often times, theres an issue with this or that feature in production

You should not have often issues with features in production. You should have them once in a while, rarely. The situation that seniors are always desperately needed to fix crisis during their vacation requires workplace that is having crisis pretty much all the time.

Capable management does not generate projects that are in crisis all the time.

Emergency sign off is correlated with a broken process in my experience.