I'm quite senior, and I don't think this is true.
Refactoring "as you go" is a no-go for a few reasons: (1) Roadmap (read: business) goals need to be reached and unless technical debt is a part of roadmap discussions -- it's usually not, because business people don't "get" code -- then it will naturally be of lower priority. (2) On a technical level, technical debt changes/re-arch should not live in feature branches. On a deeper technical level, technical debt pull requests and reviews will also usually incur a higher level of cognitive load (and thus will take longer to merge). Conflating technical debt and new features is a Bad Idea™. (3) This final point is a bit "20-20 hindsight," but with proper roadmapping and architecting (and unless you're building an MVP or proof of concept at your two-person startup), technical debt should basically not exist. Technical leaders should always push for high-quality code (even if product timelines are stretched) and technical debt is, generally speaking, a byproduct of bad technical leadership.
No matter what you do, there will be tech debt of some sort. There's no perfect solution when you unpack a problem.
My metaphor for this is a stocked product warehouse. Organizing the shelves of product and keeping it that way is obviously the right solution so people dont have to hunt when pulling. That requires someone to maintain (typically the floor manager). That's not a random schmuck you can hire for min wage to do (oddly enough). You now have a form of debt requiring you to vet a proper individual to do that job if its opened up (time spent finding someone in a smaller pool that typical temp unskilled labor). Next you have pulling with forklifts. You're supposed to hire certed folks for the job, thus a skilled worker. They cost a bit more than people who can only toss crates. But the trade off is moving pulled pallets faster. Thus you have a higher per worker cost, but higher efficiency.
Tech debt is what you're willing to trade off. Highly abstracted code in an esoteric language while super efficient and well maintained requires vetting the right devs from a very small labor pool. You lose too many of those, your business can suffer greatly if you can't fill those spots quickly. Does the project require that? Can you get away with common, off the self libraries in a mass market language and structure the code where you can take low skilled devs and ramp them up quickly or ideally need no ramp up to continue working? Over engineering, no matter how well done and beautiful is not always the answer. Like, you dont want an Formula 1 designer making everyday sedans.
Beating that dead horse, tech debt is what you're willing to put up with depending on your circumstances. Can it roadblock you? Sure. But that's just piss poor planning or forgetting the original purpose. A lot of times when tech debt is brought up, it's like the person is leaving the possibility open to place a Falcon 9 rocket on their car. It may not be that the software architecture is wrong, sometimes the idea is just stupid.
I’d like to add a caveat though. At least the places I’ve worked, tech debt is a polite and convenient way to provide political cover to all individuals.
“We made some dumb choices and I don’t want to point fingers but the more we let this go the more exponentially time consuming to maintain this will become,” and/or “some of our team can’t figure out how any of this works and we don’t want to say those people are _dumb_ exactly, but working this piece of the application could improve our ability to make modifications and generally improve the competency of our team” or maybe even “this star engineer gets obsessed with this code-shape and we don’t want to lose her, so we should let her write it ‘the right way’ to keep her happy.”
These are examples were “tech debt” is being used not necessarily for lack of understanding about the problem, but rather as a shorthand to discuss a *mostly already understood problem” while providing political cover to everyone involved. Maybe management insisted on some dumb things, maybe engineering did some dumb things, maybe if we don’t do this dumb over-engineering task we’re just going to be distracted by it for the next year.
When the “tech debt” is NOT a clear trade off being discussed using those words as a shorthand, I think it’s that there’s a lot of little issues that can come back to haunt us in a wide variety of ways that we already know how to fix (eg security, documentation, efficiency of a core process could sink us if we have unexpectedly good success, whatever, and general “risk reduction” activity that is a million little things that are too plentiful to identify in isolation with any amount of efficiency). Calling it “tech debt” in this scenario is an admission that we don’t understand the risk profile of our own code well enough to make good choices, so we’d like to spend some time organizing the code so that we 1) can understand it will enough to maintain and modify and 2) better understand the potential risks of our code (and potentially mitigate some risks along the way).
It’s sort of like, you’ve got a bunch of floor managers, but you had them all just tossing product into the warehouse as fast as possible rather than doing it in an organized way, and now they’d like to maybe do their job so that we can actually have an inventory and get what we need when we need it. Yeah, you hired floor managers, but it doesn’t magically get organized if you reprioritize them to shove everything in in whatever way is fastest. Ie, it was a dumb idea to do it that way, and maybe some of that dumb is on upper management for telling the floor managers to “just shove it in ASAP” and maybe some of that dumb is on the floor managers for not confronting upper management about it before starting, but now that we’re here, let’s just agree that there was some dumb that happened.
On point 2, you're right they should be separate, but I think the solution is, refactor to make your solution easy, then build the solution.
Point 1, if you have a significant rearchitecture to mention on your roadmap, do so. But don't make the mistake of thinking business leadership cares. You'll get farther with strong technical leaders who can push for appropriate timelines, not haggle over technical tradeoffs with executives.
Parent post is terrific and makes a lot of great points.
If you work in entprise and find bad code that's just plain old vanilla bad engineering and we should carve that out of the definition.
If you work in early stage or barely mature then the luxury of defining and scoping tech debt is something that should be treated as a necessary step to maturity. After all, whatever you think of the code or design decisions you've found, it has clearly justified itself just on basis that the company is successful enough for you to be there finding it.
I've almost never seen a situation where intentionally bad code has helped even in the short term. Maybe bad code was the best that a particular engineer could do, and nobody else was available to do it, so that made it acceptable. But any engineer writing code less well than they are able to has never led to anything good in my experience.
Even in 45 minute interviews, if I take shortcuts, I usually end up creating more problems for myself than if I had just written code the right way in the first place.
Don’t kid yourself. That is technical debt, it was our fault.
Technical debt isn’t “bad code” it’s what stops you from shipping features; and it’s more important for startups to manage, not less.
What youre referring to is very early stage stuff, where the total LoC is so trivial that a few people can just remember all the details of it in their heads, so it doesn’t matter how messy it is.
...true, but there are finite physical limits to humans; you can only keep doing that for so long before you flub it up, and lose a lot of money for your employer.
This is one of those moments that people just flat out fail to have intuition about.
Same process, same results right?
Nope.
Same process with diminishing returns <— reality of software engineering over time.
So specifically:
> After all, whatever you think of the code or design decisions you've found, it has clearly justified itself just on basis that the company is successful enough for you to be there finding it.
This is wrong.
It was the correct decision/code at the time for the scale at the time, which, since you are there looking at it, it is probably no longer suitable for.
Not intuitive is it?
Software unlike physical stuff doesn't rot/depreciate, the world instead changes around it while it stays constant. Business context and requirements change around it changes to the point where the software itself isn't as useful anymore and it has to be contorted to do something it was never designed to do over time. That's it.
In tech where things the world can move fast this is particularly true.
For example one bit of advice I've heard is that things like unit tests and a CI pipeline are a luxury for early-stage startups. An experienced developer would probably spend a day or two setting things up with, say, Heroku and a Gitlab pipeline, and at least write some unit tests for core business functionality and a few tripwire tests for the HTTP endpoints. They've done the same thing a few times and could probably just pull in a starter template for 80% of that work. Very little time spent with zero impact on velocity but lays a good foundation for later.
Another junior dev tell is getting power hungry with the freedom of a greenfield project. For example, rather than going with the boring but stable PHP framework they used at the last company, they want to do all the cool things like serverless or JS framework du jour or Kubernetes or GraphQL, depending on whatever Medium or Hacker News post they last read. An experienced dev will use "boring" technology they are familiar with as much as possible, and adopt or write new tech when there is an explicit need. For example, if your startup's secret sauce is an ML algorithm connected to an HTTP API and web app, you want to focus on that ML algorithm and tweaking and optimizing that, not figuring out how to get authentication to work with your serverless/graphql/svelte cool stack.
Making code changes means one takes on the risk of breaking things. I think the people who don't break stuff are the ones who continue to refactor. The ones who don't either are scared of breaking stuff or had a bad experience with breaking stuff.
I suspect the people who refactor have what can be describe as perfectionism or obsessive compulsive.
I'm in that bucket. I can see certain code which is too complex or didn't handle a requirement well and I want to get it to be stated more simply.
You either didn't fully read my post, or didn't understand it. It's not that you can't, it's that it's a bad idea. I would instantly reject any pull request that wasn't compartmentalized to the issue at hand. It's an organizational nightmare to handle both features as well as refactoring in the same PR.
But obviously to each their own.
What would happen if you said you need another developer to hit that delivery goal. Good chance you'd get it, right? A good EM should be able to articulate the value from non-functional work and refactoring.
The former might be a big deal for a junior but can easily be paid off with pocket money by a senior (refactored on the go). While the later more naturally should be part of roadmap and business decisions, even for a senior.
To clarify the different levels, first could be something trivial as not following naming standards, missing unit-test or using global variables instead of DI, and the bigger debts would be whole stack depending on some vendor cloud lock-in.
Both of those can be anchors that can assure quality and coherence, but they can also become anchors that prevent the ship from moving.
Documentation is a huge pain for me. I hate writing it (but I'm actually fairly good at it). Code documentation is something that I've inculcated into my personal process[0], but "product docs" can be a whole different story (not just things like PDFs, but also support sites, etc.).
The biggest problem with both documentation and unit test suites, is that they can easily become "concrete galoshes"[1], that add a great deal of work to a lot of product modification and improvement strategies.
Can we call something that prevents or makes extension and modification more difficult "tech debt"? Maybe, maybe not.
It's definitely possible to write unit tests in a fashion that helps to modify codebases, but that takes a fair bit of planning and care; not something that our industry is known for, when we are in a "ship crunch" (I have known many organizations that deliberately try to maintain a constant state of "ship crunch," because their managers believe it to be more productive). I tend to prefer test harnesses, over unit tests, because of this[2]. In fact, just this morning, I was lamenting that I need to make some changes to a heavily unit-tested codebase that may cause issues with the unit tests. The fixes need to happen, even if they pooch the tests, but I will need to go back and re-run the tests, and fix any that erroneously fail. It's a really big job, for this codebase. My testing code is generally much larger than the code under test.
[0] https://littlegreenviper.com/miscellany/leaving-a-legacy/
[1] https://littlegreenviper.com/miscellany/concrete-galoshes/
[2] https://littlegreenviper.com/miscellany/testing-harness-vs-u...
Some projects don't warrant over engineering because the requirements are often not clear in the beginning. There's a cost in terms of resources and opportunity which often doesn't make sense to incur at an earlier stage of product development.
As the first few versions play out in the market, the requirements are better understood, the product starts fitting the market better. Obviously, a lot of the earlier choices don't seem optimal for what the product becomes at a later time. Yes, theoretically, "it could have been done better", but nobody could have known for sure. It should simply be refactored so that the design of the software better fits what it is doing at a given time... and this should be an expected way to develop products.
I think "technical debt" is often the wrong term for what is going on. I'm sure there is a better analogy in terms of business and finance to what is happening... but I'm not well-versed in finance to say what it is.
Now the people who wrote those stacks and who understand them are retiring/quitting. Kids coming out of school don't want to learn these systems, nor do people off the streets. You can only pay people to come out of retirement so many times to keep the plant running. This is above and beyond mainframes, and is intertwined deep in the code that powers every single application that runs the plant today.
We can't run off the shelf software on-prem, a huge level of customization is needed to bring it in.
We cannot pivot quickly to new things or support new languages.
We really struggle to add new features/releases and add new software to drive revenue. The IT overhead that just goes into keeping the plant running every day is astounding.
This is what I think of when I hear technical debt.
Going down this road did give us advantages for a long time, but now we're in an enormous crisis. It's not an insurmountable challenge, but I would be surprised if there aren't a lot of large companies who are brought down by their technical debt as faster moving competitors move around them. I certainly feel that unless we get our act together, we will be disrupted.
Code that has worked for 30 years is more technical debt than the ability to support 'new languages' or 'pivot' the code? Maybe I am an old fart, but the opposite seems right to me.
Large code bases are difficult to add functionality to.
A code base that is easy to pivot and switch languages sounds more like a nightmare. Isn't it more likely that your new, zeitgeist-capable software would torpedo development in far less than 30 years. As I understand "new programmers", the 'reinventing the wheel' speed is measured in a few years not decades now.
Or did I misunderstand you?
It's basically paralyzing. It's so hard to get a release done, add capacity, or add new features for our lines of business (we have dozens!).
How aware are the execs about the situation? Or is it primarily the engineers, maybe technical managers, who can see the problem currently?
(Is it ok if I ask what's your job role?)
I can tell you that execs are very aware of the problem. Higher ups have spoken about it at townhalls, though they use softer language than I do. Since 2017 a lot of modernization attempts have been made(go cloud, use standards, use off the shelf software as much as possible), with very little to show for it so far.
Obviously it isn't all just tech that causes this, culture has a big part to do with it.
It feels like the scenario in the phoenix project almost, it'd be funny if it wasn't so serious.
Some of the things you mention are red flags though at least to me having worked in them before - for me they normally make me question the companies management. The biggest one seen in a previous place I worked IMO - is buy vs build as much as possible off the shelf. How many successful tech first companies who have disrupted actually use that model for their core platform? The companies I've seen get away with it until they face competitive pressure OR technology isn't their primary advantage. In fact as you get to a certain size it can make sense to build your own and reduce your vendor count + ongoing costs and take advantage of your economies of scale. How many big tech firms rewrite db's, parts, dev-ops tools or are at least open to when the advantage is there? The successful tech first companies often do, even if they were once things like book stores where it "wasn't their core business". They even open source their components often to support their business and give their tech people more cred; allowing them to attract even more tech talent. The most successful/nice to work tech places usually err to building when it concerns their platform, with some pragmatism thrown in to use modern tooling from elsewhere if required normally open source but can be bought if it offers nothing of differentiation (e.g. cloud products, databases, etc).
Cloud is just a potential enabler IMO; you still need the culture to execute. A big corporation has a lot of interacting requirements, and needs the long term flexibility to change it without being on the hook in a vendor's backlog competing with other firms. It also potentially leaks your roadmap to other competitors. Common business software (e.g. document writing, email, chat, etc) are the exception; if your a big corp your usually in a monopoly/oligopoly position - there aren't too many people doing what you do at the scale you are and most vendor solutions are really just "outsourced builds"; where the long term flexibility as the vendor pivots/changes deteriorates.
Technical debt is the idea that just like you'd take a loan for your business to grow faster, you can take shortcuts while coding your first versions (for eg. not having adequate amount of test coverage, not making it modular or extendible etc.) to get the product out or meet a deadline. Just like your business loan, you get into technical debt knowing that you will eventually pay it back (i.e. write additional tests, refactor etc.).
Not every software problem is technical debt.
While your problem seems much larger, the same problem happens in smaller scales in every software company when engineers quit and leave a complicated codebase behind for new hires to take over. You should be looking specifically for people who have experience working with legacy code in your next hiring round.
Most engineers have an eye for quality and maintainability which is great, however what they sometimes fail to see is that in a commercial reality, rapidly iterating on features to be first mover advantage is "worth" the piles of dirty dishes incurred, even if it's a huge headache later on maintaining it.
Often it's an existential threat - you either move fast (and often do the 95% solution to ship) or you are gone.
I find quite often engineers without the big picture/reality of the business can often forget that everything is an engineering tradeoff and maybe copy+paste and a bit of spaghetti is your friend today because your firm just expanded.
All of the previous analogies about libraries and dishes need an added layer: if they shut down for a week at the start to re-organise and clean, perhaps they would have gone bankrupt and the stupid solution may have actually been the best one all along.
Exactly. It's such a loaded term and it can almost always be replaced by saying something specific.
I think the counter argument is that not everything needs to get turned into a negotiation around justifying business value. Each developer should just by default get 25% of the year to improve the code they're responsible for. If a developer wants to spend 6 or 9 months one year doing some big refactor then that's one thing, but it's not a good work environment if there needs to be weeks of meetings every time someone wants to fix deprecations so that they can upgrade some library or whatever.
A generic term like “tech debt” is lazy and does not capture anything about the business value or decision making process that led to the code or would lead you to want to replace it.
For example in many organizations you make a completely justified business decision to put less time or more junior engineers on a project that isn’t that important. This isn’t tech debt, it’s a conscious decision made by completely rational people.
On the flip side, if that project that was of lesser importance becomes important again you can make another completely rational decision to go in and clean it up to make it faster/more reliable/simpler or whatever the goal may be.
In my (somewhat limited) experience the kind of value that you get from reducing technical debt is not easy to explain to the business side, because it doesn't immediately translate into new features.
> And then the other issue is that a huge amount of technical debt is just code that a more senior engineer would refactor as they went.
A limited amount of spaghetti can be untangled during the normal development process, but I don't think significant changes in design can be done while working on features. Once you get to that point, inevitably you will have to set time aside specifically for dealing with tech debt.
Not to mention that the kind of organizations that accumulate tech debt rapidly tend to also complain if you spend more than the minimal amount of time necessary on a feature. "It took you, a mid-level dev, 3 days to finish a feature that a starter could complete in a day by copy-pasting existing code with some SO stuff? You must not be too good at your job."
This is because, regardless of whether people agree on its definition, most of the things that fall under "technical debt" incur indirect or delayed costs for the business.
The managers have, at some point, a choice: We have $X. We can either spend some percentage on churning out products that directly and immediately make us money. Or we can divide it between that, and maintenance activities.
Choosing to neglect (or underfund) the maintenance activities doesn't cost the business anything for some period of time (the delay). And performing them does not (within the period of that delay) bring in any revenue. It is only a cost to them, it has no visible upside within their time horizon. By the time the costs start catching up, it's death by a thousand cuts. Usually they'll go from 100/0 dev/maintenance to 99/1, then 98/2, and so on. Then one day it flips from 80/20 to 20/80 and they realized they fucked up. But by then they've been promoted, so who cares?
You have to convince them, and perhaps they'll only learn by being burned (but I doubt it), that they should extend their time horizon by a month or a year so they can see the consequence of this decision. I've never succeeded in this.
At my previous job we were experiencing a massive number of retirements. Literally centuries of experience leaving every month (aggregated across all teams). No hiring process, without comprehensive training, could possibly keep up with this loss. But the retirements were always (for each team) just far enough over the horizon that they didn't realize the issues until after the retirement and the new hire floundered (due to lack of training/mentorship).
Pretty much what I noticed as well.
> You have to convince them
This is exactly the part that I hate. Good practices, code quality, design etc. are not a hobby of mine. The business is not doing me a personal favor by allowing me to practice these ideas.
The reality is it's all trade-offs and sometimes those debts never need to be paid at all. "Bad code" that is reliable within the constraints it is exercised and that is isolated may appear to have to tech debt, as in "if we changed the design, we could support XYZ feature" but if XYZ feature never materializes, it's a waste of time to touch it.
Maybe we can come up with more appropriate language for this? Something akin to "inflexible" code. Calling something "debt" assigns too much subjective value to it while making it seem objective.
In most cases, the concept doesn't really clarify anything, but people don't seem to know how to translate 'our' problems into something that is actually meaningful for the other people in the room.
The term 'technical debt' just obfuscates what is really going on. It doesn't provide any real insight.
You may like the blog post I linked in another post here.
See: http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_... [pdf] for an example of an analysis of a technical debt analog in other industries.
Does that sound acceptable?
It's a little wild to imaging buying a house without incurring some debt. This is normal and OK. You don't go crazy with it, though, like getting a loan for 20% over the mortgage with a 12% interest rate. Same with technical debt: cut a few corners if it lets you go to market more quickly and start earning income before your competitors can launch, but not at the cost of utter spaghetti code written in MUMPS or something. Your class hierarchy isn't perfect? Fine - your customers aren't paying you for that anyway.
So go ahead and borrow. Just don't go so far in debt that paying it off becomes burdensome.
I had a notion that this can never get so big that you can't unravel it, but I think it's a matter of how much power the problem people have. I've seen one project that will be broken forever because, while they've gotten a couple of the biggest architectural astronauts away from it, one of the most prolific will never leave, and does not accept that just because you can do something doesn't mean you should do it.
I think this is an excellent point, but we still need a shorthand way to talk about it.
> They'd say instead something like "I was thinking that if we changed X we would get value Y."
The problem I have with this is it's very hard to make a compelling argument, and without one, nothing gets done. A lot of the time technical debt is tantamount to a slow death by a thousand cuts, and "value Y" is more like "save some unknown but non-zero amount of time in the future".
One example is when you have (tens of) thousands of lines of code that don't have unit tests, have high coupling, and no or outdated documentation. Maybe there's some known bugs, but they're of low importance (in other words: fixing them is low value to the business). That code works today, and if not modified it'll continue to work. If it does eventually get modified, then there are no guarantees: maybe the modifications won't break anything, or maybe they'll cause dozens of new bugs.
If we refactor today and add lots of tests, it'll probably mean that future modification causes fewer bugs (saving time/money), but we actually can't guarantee anything -- aside from the refactor will cost a bunch of time, now.
> a huge amount of technical debt is just code that a more senior engineer would refactor as they went
In a case like I described above you can sometimes refactor as you go but it's been my experience this is often a deep rabbit hole. Sometimes your refactor requires touching a dozen layers and related bits and before you know it it can easily be several orders of magnitude more work and more risk vs the "quick fix".
Another thing I often run into that we describe as "technical debt" is about foundational designs that are (currently) wrong. This includes things like database schema and core structure of the application. I say "currently" because some where not clearly wrong at the time they were made, but it had become clear since then and yet more stuff was still built on top.
As an example, I'm working on an application that was built to run on a single instance but we'd like to be able to scale horizontally. One of the problems is the application uses what is effectively an in-process cache for a lot of the database objects. The objects are not pure DTO-style objects and so the cache can't just be moved to external (Redis or something). The schema is not that well designed, so entirely removing the cache would almost certainly kill performance, and might mean refactoring half the code anyway. A layer up, several parts of the app (including UI forms and some background processing) are built assuming they have a consistent view of things, and ignoring these issues causes all kinds of strange consistency problems as data is silently overwritten/changed.
That type of problem is not at all fixable by "refactoring as we go". In some sense I'd love to do a totally clean rewrite, but that just isn't going to happen largely because the business has no appetite for that. In the past I've done the massive rewrite thing before, and going a couple years without delivering anything from it is crazy stressful and no one is happy.
Instead, we're working on rewriting major (but approachable) chunks of this one at a time, while retaining compatibility with other parts of the system and still trying to find things that provide customer/business value as part of it (either fixing long-standing bugs or introducing new features). It's hard, and overall is more work than just rewriting from scratch, but means we can deliver value as we go. Maybe this is what you mean by "refactor as we go" (or maybe that's just how I should describe what we're doing to higher-up), but to me we're basically starting from zero and at best, pulling small chunks of the old code in, so it definitely doesn't feel like "refactor".
(I know this post is explicitly about not calling this "technical debt" but I can't help but think this approach is analogous to a payment plan. :) )
Most people seem to get that the issue of technical debt is really a business issue about tradeoffs. So, if the person who is concerned about technical debt is senior enough they will have some ability to see the business tradeoff and thus can completely avoid the phrase. They'd say instead something like "I was thinking that if we changed X we would get value Y."
A person who uses the phrase Technical Debt is signaling that they aren't capable of saying something that direct.
That's one issue.
And then the other issue is that a huge amount of technical debt is just code that a more senior engineer would refactor as they went. It's a junior engineer who says, "this is a huge mess and I need a full stop of my contributions to the business so that I can focus on untangling it."
But the problem with junior engineers doing refactoring/rewriting is that they end up creating a different mess of code and bugs.
Am I right 100% of the time about this? No. But I'd love to raise for engineers here that although the phrase technical debt caries a lot of emotional weight, it also brings a lot of baggage from all the times management has heard the phrase from someone that didn't understand the bigger picture. With a little bit of work you could understand the business implications and then end up completely avoiding the use of the phrase.