The teams that drown in tech debt tend to have roadmaps that are strictly customer facing work, that can get you very far but in the end you'll stagnate in ways that are not easy to fix, technical work related to doing things right once you know exactly what you need pays off.
However, I've worked for a small company for quite a while now. We've had several successful projects and several failures.
In my experience, technical debt taken too early can easily double the time it takes you to find out if a project is a dud. That matters to us.
My general rule is: push off technical debt as late as you can. Aways leave code slightly better than you found it. Fix problems as you recognize them.
I think a big mistake developers make is thinking "make code better" should be on some roadmap. You should be making code better every time you touch it. Nothing about writing a new feature says that you have to integrate it in the sloppiest way possible.
I vehemently agree.
One of my first jobs was working for a mathematician at a bank, who could code well enough to compute his mathematical ideas, but not a software engineer so hired me to do more of the coding for his team.
He would say "Jim, just get this done but don't spend time making it fancy." In other words, don't spend time refactoring or cleaning up the code, expressed in his own words.
I would say "sure" and then proceed to refactor and clean up the code as I went. It took less time than it would have to write the code "ugly" then deal with the accumulated tech debt, and I finished everything on time so he was happy.
Credit card debt (e.g. sloppy code and test-free critical backend processes) is pretty bad and should be paid down ASAP.
Mortgage debt (e.g. no UI tests on the front-end) is quite safe and you can kick the can down the road.
I never liked the "debt" metaphor. If a housing developer neglected to build a proper foundation, would you call that "debt?" I feel like it's very similar, it's bad metaphor for a concept that has very little to do with finance.
How would you know if the poor commercial performance was due to tech debt or not though? It's the intangibility of tech debt that makes it so insidious.
My personal experience has been that tech debt is more often caused by business level decisions and not engineering decisions. Deadline on this contract is next week so let's ship what we got and worry about it later. Hey, good news everyone we just pivoted 180 degrees so let's try to salvage what we've got.
So yes it very well might be that a mountain of tech debt was the final nail in the coffin. But why was that tech debt there in the first place? I was understanding the GP as saying they saw business decisions leading to poor engineering instead of engineering just doing dumb things on their own. I've seen plenty of examples of the latter but a lot more of the former in my travels.
Not to say you're wrong, but we have no real way of even deciding. All I can do is lean on my own experience, but I've seen nowhere near every product team out there and the ones I have seen are nothing close to randomly sampled or blinded.
Another way of thinking about it is that you have N months of runway, and based on your velocity you can pull off a pivot in M months, and the more tech debt, the more time it will take to successfully pivot. If you don’t have a full pivot worth of runway remaining, and you need to pivot, you die. (Of course this oversimplifies by holding pivot magnitude equal but hopefully this illustrates the point.)
I do agree that away from the margin, companies that are incredibly successful can afford to punt harder on tech debt. I suppose “know thyself” might be useful advice here; it’s probably not good advice for the median startup to ignore tech debt completely IMO.
I think the main point though is to optimize for agility; tech debt can let you move faster in the short and even medium term, so sometimes it’s right to tactically take on some debt. But not so much that you get bogged down later; make sure you carve out time to fix the stuff that starts to be painful.
If you’re running a startup and haven’t yet found your feet in terms of a product offering, and you’re building your product(s) in such a way that technical debt builds up through continuously layering half-baked on half-baked, it’s indicative that you’re not actually pivoting and not actually evolving, you’re just adding new half-baked ideas to a half-baked system… and being able to do that at twice the speed isn’t going to address the real problem: half-baked ideas don’t make a product, whether that’s 10 half-baked ideas or 100.
My experience is that any company in which evolution/experiments/pivoting is constrained within the boundaries of what already exists because of the sunk cost fallacy has made a grave error at a leadership level, not at a code level. If you can’t validate something without mashing more code into your system, that’s the problem to address.
I’ve seen companies with horrendous tech debt die, and you could certainly frame their death as being a consequence of the tech debt (“if they had just got the perfect system…”) but that assumes the perfect system would somehow prevent them from making the mistakes that got them there in the first place. It wouldn’t. The technical debt is an expression of their mistakes, not the cause. You could dump the perfect system at their feet and they’d be surrounded by garbage again a few years from now.
They were constantly "pivoting", but leaving the old junk around.
Take Facebook: They build an empire on PHP. Now they have built some clever compilers on top on PHP in order to make it safe and performant without breaking their existing piles of legacy code. Overall this is probably ridiculously inefficient compared to just using a safe and performant platform from the beginning. But using PHP in the beginning allowed them to move fast in the critical growth phase.
If we're married to the financial debt analogy, then technical debt has compounding interest like a payday loan... and payday loans are typically used in very distressed circumstances, and are very dangerous. There's appropriate times to take a payday loan, and there's appropriate times to take on technical debt, but it has to be handled with great care and be an immediate wake up call to address the underlying cause.
The more insidious kind behaves more like a completely unhedged call option - fine, until it very suddenly isn't.
Of course it is different for a steady-state company or organization. You need to keep technical dept at a manageable level.
How many successful companies actually do this?
At one point in my career I transitioned from a startup to one that had been acquired some 12 years before, and found it to be even more chaotic than the startup. Instead of playing a frantic game of whack-a-mole with all the pivots and feature ideas of the founders, you had a few dozen teams playing whack-a-mole with the pet projects of their respective product managers who were trying to make a name for themselves. Which was much worse because you had to coordinate with every other of those teams, and of course work with all the integrations with the parent company.
Charitably speaking, maybe these older successful companies are bad simply because the field of software engineering was still too immature when they came about, and today's startups will actually pay back their debt when they become successful in the future. Sure, we have better tools now than then, but we still don't have a static analysis tool that can determine if we built the right or wrong thing for an ever changing market.
Not taking time is, at heart, where tech debt is born. That can manifest debt across all areas of the development process. Pressure to not take time can certainly come from management, but I have also witnessed on numerous occasions the reverse, where management asks developers to slow down and take the time to do the work well; sometimes to no avail.
Either way, your underlying thesis is quite true that given the perfect system an imperfect team will quickly reintroduce said problems into the system. This is why many software companies have become hyper-picky (even before the tech crash) about hiring. They want to try and avoid building a team that wants to shortcut that time.
But in the counterfactual if they'd tried really hard to avoid tech debt that would have slowed them down at the beginning, not to mention there are plenty of organizations that will write very complex abstract code to avoid tech debt, and end up making the code base incredibly painful to work with. So overall did get they get less swings?
I've worked on a lot of old code bases and the biggest issues I've run into, issues that crippled development velocity, were 95% boneheaded decisions and overengineering. And never the types of code quality issues someone like Uncle Bob talks about in Clean Code.
This is totally true, but taken too seriously it leads to inability to learn anything from almost any information whatsoever. What’s more, whatever you do (whether you take the advice of those who have gone before or not), you will not be able to decide whether you made good decisions or merely “survived”.
How does one proceed when anything can be survivorship bias and determining cause and effect for large scale operations like running a business is essentially impossible.
(When I say “anything can be survivorship bias” I specifically mean that no matter the cohort you cannot decide whether you’ve accidentally excluded unknown failures, and hence you have no assurance of the actual strength of any analysis you do).
Not my experience..
What does a “tech failure” look like? Do the servers catch fire? Is the web site down? Maybe people are unable to login to their stations?
Hi-tech business is “Tech”, so the failure of the business is in fact the tech failing. More specifically, the business was unable to direct the tech to solve real problems and solve them well enough.. New hires took too long to onboard.. Engineers were only superficially productive.. Communication between the stake holders and engineers was lacking.. etc.. etc..
Take note that in all scenarios above “work” is being done, “progress” is being made.. ceremonies are everywhere and success is seemingly around the corner.. Or is it?
It’s just very hard to see these issues, they are hidden under layers of meetings, firings, hiring, pivots, milestones with little progress in actual business value.
When there appears to be such a distinction, that's usually a manifestation of something like Conway's law, a symptom that there exists an unhealthy organizational divide between business and technology.
This doesn’t count as a tech problem?
A common theme right now is 'AAI', using people to fake an AI that may not come into being at all, let alone before your runway (inevitably) runs out.
The developers you'd hire to make it an actual AI and the developers you'd hire to make it a Mechanical Turk are very different skill sets.
That is, plan A is Stripe, plan B is Theranos.
I would think that a poor quality product, or one not as good as competitors would be a big killer. Google, Facebook, Amazon have amazingly superior products. I think you're missing something.
How about the cases where it caused fines due to failed security compliance that didn’t help the situation. Thinking fintech companies especially.
In 15 years of doing technical due diligence (200+ jobs) I have yet to come across a company where the tech was what eventually killed them. But business case problems, for instance being unaware of the true cost of fielding a product and losing money on every transaction are extremely common. Improper cost allocation, product market mismatch, wishful thinking, founder conflicts, founder-investor conflicts, relying on non-existent technology while faking it for the time being and so on have all killed quite a few companies.
Tech can be fixed, and if everything else is working fine there will be budget to do so. These other issues usually can't be fixed, no matter what the budget.