News for hackers. Stuff that matters.
Re:The absolute worst scenario happened by JKCalhoun on Monday March 22, 2021 @11:00AM (#26541770)
Is there another industry where a few employees with "domain knowledge" can sink a company?

Asking because, while, yes, throwing money at the problem might be a solution, I think the obvious bigger issue is how does one allow themselves to get into this situation?

Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.

Re:The absolute worst scenario happened by gregmac on Monday March 22, 2021 @12:21PM (#26542941)
At the highest level, this is a leadership issue. There's a lack of someone at a sufficiently high level (eg: C-suite) that properly understands technology, and that's probably caused by the rest of the leadership team not understanding their limitations.

At a lower level, it's allowing individuals to become knowledge silos (or having a bus factor [1] of 1); lack of documentation; not having+testing disaster recovery plans; neglecting systems, code bases, and platforms; letting technical debt accrue endlessly (or worse: not even realize that's what you're doing).

How to avoid? Make sure there's a strong, competent technology leader with a vested interest in the company succeeding, and empower them to make the changes they need to.

What should that person be doing? Making sure all the stuff I just said doesn't happen.

[1] https://en.m.wikipedia.org/wiki/Bus_factor

Re:The absolute worst scenario happened by foobiekr on Monday March 22, 2021 @03:01PM (#26545231)
There are a LOT of industries with this problem. Specialist stuff like military shipbuilding is an example, but also things like precision ball bearings. Knowledge gets lost all the time and old timers have to be brought back in after the fact. Or the technology is just lost and a replacement has to be reinvented (famously, the production of tamper material for the US thermonuclear nuclear weapons).
Re:The absolute worst scenario happened by shagie on Monday March 22, 2021 @03:20PM (#26545445)
The precision ball bearings thing reminded me of a news item from a few years ago: https://www.washingtonpost.com/news/worldviews/wp/2017/01/18...

The relevant part of the article:

> To anyone outside of the ballpoint pen manufacturing world, it might seem hard to understand what, exactly, is so surprising about this development. China already produces 38 billion ballpoint pens a year, according to China Daily, which is about 80 percent of all ballpoint pens in the world. That's a lot of pens, but there was a catch: China had long been unable to produce a high-quality version of the most important part of the pen, its tip.

> The tip of a ballpoint pen is what makes it a ballpoint pen. At the tip, a freely rotating ball is held in a small socket which connects it to an ink reservoir that allows the pen to write or draw lines. Manufacturing a ballpoint pen tip that can write comfortably for a long period of time requires high-precision machinery and precisely thin steel, but for years China was unable to match those crafted by foreign companies.

Re:The absolute worst scenario happened by ufmace on Monday March 22, 2021 @02:53PM (#26545098)
> Or rather, how does one avoid getting into this situation.

IMO, the problem is technical debt. Most likely, in the majority of these cases, the engineers working on the system knew perfectly well that it was a mess, was very difficult to learn how to maintain, and many parts were dangerous to modify. Most likely they told management multiple times that this was a problem and what the consequences of it would be. They would have refused to budget resources for improving it, so everyone either saw the writing on the wall and quit, or gutted it out and learned how to manage the system eventually. Maybe the system limps along for a while longer, but it's only getting worse and the knowledge needed to maintain it more siloed and specialized. Eventually you lose a key person, something breaks that nobody left can figure out, or maybe the whole thing just implodes like the main story here.

To avoid it, you must understand the pattern, hire good technical people, and give them the resources needed to improve the system in an orderly fashion.

Re:The absolute worst scenario happened by hunter-2 on Monday March 22, 2021 @01:15PM (#26543681)
>Is there another industry where a few employees with "domain knowledge" can sink a company?

I guess any company where leadership got there without progress on the technical side. Law firm partners get there after being lawyers. Consultant partners get there after consulting.

But if you are the CEO of an IT company without ever having to code, then you may never know the things you must take care to prevent a washout.

Re:The absolute worst scenario happened by 908B64B197 on Monday March 22, 2021 @07:12PM (#26548725)
> Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.

Mentorship and documentation.

You need to set aside time and resources for these two things, and make contributing to them worth something in the performance review.

If all you do is count JIRA tickets closed, you'll forever end up in this situation.

Re:The absolute worst scenario happened by ISL on Monday March 22, 2021 @11:02AM (#26541811)
> Or rather, how does one avoid getting into this situation.

Documentation?

Re:The absolute worst scenario happened by giantrobot on Monday March 22, 2021 @12:16PM (#26542871)
The issue with "documentation" is there are never enough resources devoted to it at most companies. When given a schedule it only covers the main project goal(s) and no time for documentation or refactoring or any quality of life improvements.

Significant documentation is also not something easily written in slack time. With any non-trivial project good documentation is a full time effort. It also needs to be updated as the project evolves lest it become out of date and incongruous with the behavior of the system.

There's dreams of in-line documentation but that's only going to cover individual methods and classes and not necessarily whole modules or major subsystems of a project. It's also not going to necessarily cover in-line documentation from other parts of a project in different languages e.g. comments in config files.

There's lots of technical ways to help documentation but there's no replacement for documentation being a high level project goal with resources allotted to it.

Re:The absolute worst scenario happened by gregmac on Monday March 22, 2021 @12:37PM (#26543171)
Part of a way around this is broadening your definition of "documentation".

For example, in code: good naming, simple structure, unit tests, and useful comments all can count. A set of easy-to-understand unit tests that cover real scenarios is much better than even the greatest documentation ever written.

A fully scripted build and ideally also deployment (CI/CD pipeline) beats a checklist with tens of items that have to manually followed.

An issue system with good descriptions of changes (including why and what), reproduction steps, expected outcomes, links to related issues, etc also can go a long way.

All of this stuff has the side effect of making day-to-day work simpler, overall, while actually just being a part of that work.

Re:The absolute worst scenario happened by giantrobot on Monday March 22, 2021 @06:00PM (#26547863)
> For example, in code: good naming, simple structure, unit tests, and useful comments all can count. A set of easy-to-understand unit tests that cover real scenarios is much better than even the greatest documentation ever written.

1. Naming things is hard. It's a bit of a joke but good descriptive yet usable naming is hard. It can get harder after refactoring or adding in features you didn't expect the last time you were in the module.

2. Project structures start out simple. Then you need to add some feature or fix some major issue with a due date of last week. As a project gets bigger the simple structure of yesterday might not serve the needs of today. Resources for refactoring are usually somewhere around the priority level of "Good Documentation".

3. Comments need maintenance like explicit documentation. Tight schedules lead to "# TODO add documentation here". Then you get the assholes that believe comments or in-line documentation is something you put in check in notes.

4. A "unit test" that covers a "real scenario" is not a damned unit test. It's an integration test (or however you label it) and it's not likely something your CI/CD system can run on the regular. If a real scenario is touching a production-load DB or something it's a big deal to run the test. Tests like that need resources to run and non-trivial resources to write. You're not just making sure a method returns a float or throws an error correctly.

I'm not saying your suggestions are bad or shouldn't be used. The issue is management rarely sees any value with documentation, tests, or even code cleanliness. They're rarely incentivized to care about those things so they don't. They get their bonus on feature checkboxes or hitting a deadline. At the end of a project they'll schedule a "documentation" sprint or some dumbass thing that accomplished little useful output. It's worse when you've got a "move fast break things" chanting moron doing the scheduling.

Re:The absolute worst scenario happened by flukus on Monday March 22, 2021 @05:02PM (#26546993)
It doesn't help that "Technical Writer" is now an incredibly rare job title, it was one of the first in a long line of specializations that were pushed on to general developers. As you mentioned, they were never allocated the extra time but also don't have the training and experience that technical writers once did. It was also a very poor target for outsourcing.

The specializations common on a team even 20 years ago was much closure to the "Surgical Team" model from the mythical man month than what we have today.

Re:The absolute worst scenario happened by rantwasp on Monday March 22, 2021 @11:36AM (#26542312)
nah. documentation only goes so far.

you handle this by having and exercising a disaster recovery plan and having (and listening to) competent people that would spot this issue in a jiffy.

Re:The absolute worst scenario happened by foobiekr on Monday March 22, 2021 @03:03PM (#26545256)
This. I actually think just documentation is, well not completely worthless but close to. Imagine telling someone they could learn to weld airframes by reading. Just no.
Re:The absolute worst scenario happened by twobitshifter on Monday March 22, 2021 @04:23PM (#26546394)
For software, it’s the maintenance task that’s helped by documentation not building. They still need to know how to build but it’s a lot easier with documentation. I know that when we use software packages we all rely on the documentation to understand how to use it and code our desired behaviors. It’s obvious that our own code can benefit from documentation in the same way.
Re:The absolute worst scenario happened by rantwasp on Monday March 22, 2021 @05:17PM (#26547236)
documentation is required but not sufficient. also, documentation is usually outdated really fast if the person maintaining the system leaves and/or does not care