Now the people who wrote those stacks and who understand them are retiring/quitting. Kids coming out of school don't want to learn these systems, nor do people off the streets. You can only pay people to come out of retirement so many times to keep the plant running. This is above and beyond mainframes, and is intertwined deep in the code that powers every single application that runs the plant today.
We can't run off the shelf software on-prem, a huge level of customization is needed to bring it in.
We cannot pivot quickly to new things or support new languages.
We really struggle to add new features/releases and add new software to drive revenue. The IT overhead that just goes into keeping the plant running every day is astounding.
This is what I think of when I hear technical debt.
Going down this road did give us advantages for a long time, but now we're in an enormous crisis. It's not an insurmountable challenge, but I would be surprised if there aren't a lot of large companies who are brought down by their technical debt as faster moving competitors move around them. I certainly feel that unless we get our act together, we will be disrupted.
Code that has worked for 30 years is more technical debt than the ability to support 'new languages' or 'pivot' the code? Maybe I am an old fart, but the opposite seems right to me.
Large code bases are difficult to add functionality to.
A code base that is easy to pivot and switch languages sounds more like a nightmare. Isn't it more likely that your new, zeitgeist-capable software would torpedo development in far less than 30 years. As I understand "new programmers", the 'reinventing the wheel' speed is measured in a few years not decades now.
Or did I misunderstand you?
It's basically paralyzing. It's so hard to get a release done, add capacity, or add new features for our lines of business (we have dozens!).
How aware are the execs about the situation? Or is it primarily the engineers, maybe technical managers, who can see the problem currently?
(Is it ok if I ask what's your job role?)
I can tell you that execs are very aware of the problem. Higher ups have spoken about it at townhalls, though they use softer language than I do. Since 2017 a lot of modernization attempts have been made(go cloud, use standards, use off the shelf software as much as possible), with very little to show for it so far.
Obviously it isn't all just tech that causes this, culture has a big part to do with it.
It feels like the scenario in the phoenix project almost, it'd be funny if it wasn't so serious.
Some of the things you mention are red flags though at least to me having worked in them before - for me they normally make me question the companies management. The biggest one seen in a previous place I worked IMO - is buy vs build as much as possible off the shelf. How many successful tech first companies who have disrupted actually use that model for their core platform? The companies I've seen get away with it until they face competitive pressure OR technology isn't their primary advantage. In fact as you get to a certain size it can make sense to build your own and reduce your vendor count + ongoing costs and take advantage of your economies of scale. How many big tech firms rewrite db's, parts, dev-ops tools or are at least open to when the advantage is there? The successful tech first companies often do, even if they were once things like book stores where it "wasn't their core business". They even open source their components often to support their business and give their tech people more cred; allowing them to attract even more tech talent. The most successful/nice to work tech places usually err to building when it concerns their platform, with some pragmatism thrown in to use modern tooling from elsewhere if required normally open source but can be bought if it offers nothing of differentiation (e.g. cloud products, databases, etc).
Cloud is just a potential enabler IMO; you still need the culture to execute. A big corporation has a lot of interacting requirements, and needs the long term flexibility to change it without being on the hook in a vendor's backlog competing with other firms. It also potentially leaks your roadmap to other competitors. Common business software (e.g. document writing, email, chat, etc) are the exception; if your a big corp your usually in a monopoly/oligopoly position - there aren't too many people doing what you do at the scale you are and most vendor solutions are really just "outsourced builds"; where the long term flexibility as the vendor pivots/changes deteriorates.
Technical debt is the idea that just like you'd take a loan for your business to grow faster, you can take shortcuts while coding your first versions (for eg. not having adequate amount of test coverage, not making it modular or extendible etc.) to get the product out or meet a deadline. Just like your business loan, you get into technical debt knowing that you will eventually pay it back (i.e. write additional tests, refactor etc.).
Not every software problem is technical debt.
While your problem seems much larger, the same problem happens in smaller scales in every software company when engineers quit and leave a complicated codebase behind for new hires to take over. You should be looking specifically for people who have experience working with legacy code in your next hiring round.
Most engineers have an eye for quality and maintainability which is great, however what they sometimes fail to see is that in a commercial reality, rapidly iterating on features to be first mover advantage is "worth" the piles of dirty dishes incurred, even if it's a huge headache later on maintaining it.
Often it's an existential threat - you either move fast (and often do the 95% solution to ship) or you are gone.
I find quite often engineers without the big picture/reality of the business can often forget that everything is an engineering tradeoff and maybe copy+paste and a bit of spaghetti is your friend today because your firm just expanded.
All of the previous analogies about libraries and dishes need an added layer: if they shut down for a week at the start to re-organise and clean, perhaps they would have gone bankrupt and the stupid solution may have actually been the best one all along.
Exactly. It's such a loaded term and it can almost always be replaced by saying something specific.
Some projects don't warrant over engineering because the requirements are often not clear in the beginning. There's a cost in terms of resources and opportunity which often doesn't make sense to incur at an earlier stage of product development.
As the first few versions play out in the market, the requirements are better understood, the product starts fitting the market better. Obviously, a lot of the earlier choices don't seem optimal for what the product becomes at a later time. Yes, theoretically, "it could have been done better", but nobody could have known for sure. It should simply be refactored so that the design of the software better fits what it is doing at a given time... and this should be an expected way to develop products.
I think "technical debt" is often the wrong term for what is going on. I'm sure there is a better analogy in terms of business and finance to what is happening... but I'm not well-versed in finance to say what it is.