One is the "big re-write". They start on a new code base, and try to develop it in parallel. It takes a very long time, and the teams have to work on two solutions for some time. It's a big bang approach, and it often fails, or drags on for years.
The second is massive refactoring. It requires intensive testing and best practices. This strategy requires that the teams focus on testing intensely. However, often the testing culture is not there, which is why the code became unmanageable in the first place. It's kind of like starting over. And the new focus and discipline on testing is hard for teams to do without strong leadership, training, or new talent.
The last, and most effective in my opinion, is to go with a service-based, incremental approach. If the code base is not already using services, APIs must be built. Frontend/apps must be de-coupled from the legacy components. A clear domain model has to be agreed upon, and then parts of the legacy codebase are put behind APIs, and de-coupled from the rest. Over time, sections are refactored independently, and the APIs can hide legacy away. Maybe the legacy parts are refactored and replaced, or they stay around for a while. But the key is, that this approach allows multiple people or teams to work in parallel and focus on their areas. This is domain-driven design in action, and it works. New features can actually be developed sooner, even though the legacy is not replaced yet.
In the end, overcoming tech debt is about people. And on larger code bases, it's more of an organizational problem than a code problem. Developers need to be able to move forward without having to navigate too much code or too many different people.
I do break my infrastructure apart, but far less aggressively than some advocates.
I'm curious what 25 years had lead you to believe.
I very much believe in microservices. I've repeatedly seen small library based approaches fail because one of the key and near universal truths of tech debt IME is code being tightly coupled. When you force an API interaction to happen via an outside protocol, you force a clean contract and a culture of coding to a contract. Decoupling the code allows yet team to move faster and more independently.
I was once handed a project where the original architect drank the microservices kool aid. It didn't stop the different services from being tightly coupled to one another - it just made the pain of that happening worse.
It made testing a pain - you needed to set up and run 11 different services on your machine to test anything.
It made debugging a pain - you had to trace calls over multiple different services with code often written in different languages. Debugging became more and more like detective work.
It created a multiplicity of irritating edge cases. The 'calculation server' could time out if it took too long - and it sometimes did. Serialization/deserialization was also an area rife with bugs.
The code quality got worse due to this approach, exacerbated by the team lead at one point giving people 'responsibility' for different services to different people.
I think microservices where they've "worked" has typically been a path of least resistance to realizing Conway's law - a tacit acknowledgement that different corporate fiefdoms want to write and deploy code in their own way and won't communicate effectively with one another. In that respect I think it's effective because it's easier to draw up REST API contracts between disparate often different-language-speaking teams using microservices than it is to draw up library API contracts.
Surrounding technical debt with integration tests and incrementally refactoring (decoupling modules, deduplicating code and adding assertions) is the only way to approach technical debt.
>In the end, overcoming tech debt is about people.
No. It's a technical problem. People problems definitely exacerbate it - deadlines, politics, etc. but it's still a technical problem in the end.
That is explicitly not the "microservices kool aid." The first thing a microservice needs to be is independently testable, so it can be independently developed.
All you had there was one monolithic service.
It definitely is a drag, but there's just no way around that.
1. Tell your team that cleaning up the tech debt is a major company priority, and even if some mistakes are made along the way, that's an acceptable price to pay for the benefits involved. People shouldn't let the fear of breaking something, dissuade them from cleaning up the mess.
2. Have your team invest heavily on building test/QA infrastructure, so that if they were to break something, it would get automatically caught and flagged, before it reaches production. If the advice given in bullet 1 scares you, then you need to double down here to make up for it.
I've been in teams where pull requests that significantly cleaned up the code base were literally rejected, because people were paranoid that any change at all could break something in unforeseen ways. People just bunkered up into a "if it ain't broke, don't touch it" mentality, which meant that the tech debt problem never ever improved. Ultimately, the only way to get yourself out of this hole, is by encouraging people to take risks even if it involves them sometimes failing, and building better safety nets to catch them on the occasions when they do fail.
Was this in a project with or without unit tests? Usually you can win over people like this by writing a test harness around the area affected, getting that through their desks, and then pushing for change.
If even that is rejected you take it upstairs and make an ultimatum or straight up quit and pat yourself in the back for a job well done, regardless of which one you picked.
Sometimes the unit tests are the technical debt...
Unit tests rarely contribute much to the technical debt, because worst generally-seen case is that you can just throw them away. I've seen that, unit tests so bad they weren't usable for much. I'm yet to see a code base destroyed by trying to make it testable, whereas I've seen a ton of codebases that were made quite powerful and flexible, yet also fairly pleasant to use, because they were built to be testable.
Also, in the end I'm far more interested in "automated" testing than anybody's pedantic definitions of what "unit" or "integration" or whatever testing is. That's not the point. The point is that I ought to be able to set up an automated built server and run useful, meaningful tests on it.
Then anyone who writes a test has to make it pass, and anyone who makes a change that breaks a test has to fix it. No rotting tests that passed long ago and have been failing for months that everyone ignores because whatever
Worst codebase I have ever seen. The incumbents had zero interest in paying down technical debt. Unit tests were just another thing that could be used to game metrics and justify budgets.
Metaphorically speaking, the group issued predatory high-interest payday loans to keep the customer paying increasing amounts of interest on a growing amount of technical debt, forever.
Unit tests are by their very nature tests that are tightly coupled to your code.
Tight coupling is usually the biggest component of your technical debt.
You want tests that are more loosely coupled - integration tests.
That's why you need to surround the code with realistic integration tests first before changing anything.
>Ultimately, the only way to get yourself out of this hole, is by encouraging people to take risks
No, that's absolutely wrong. You need to first de-risk changes by creating a test harness that catches the bugs that terrify your developers.
Unfortunately big test suites won't help for this type of refactoring though, on the contrary they might even be in your way.
I believe the basic approach is to write tests to capture the current behaviour at the system boundaries - for a web application, this might take the form of automated end-to-end tests (Selenium WebDriver) - then, progressively refactor and unit test components and code paths. By the end of the process, you'll end up with a comprehensive regression suite, giving developers the confidence to make changes with impunity - whether that's refactoring to eliminate more technical debt and speed up development, or adding features to fulfill business needs.
This way, you can take a gradual, iterative approach to cleaning up the system, which should boost morale (a little bit of progress made every iteration), and minimises risk (you're not replacing an entire system at once).
I've used this approach to rewrite a Node.js API that was tightly coupled to MongoDB, and migrated it to PostgreSQL.
[0] https://www.amazon.com/Working-Effectively-Legacy-Michael-Fe...
Personally I don't mind having to deal with the technical debt, but I absolutely detest being forced into technical debit. This is especially true if I have not yet been allowed to deal with what ever debt that already exists in a given system.
The worst thing to hear in a meeting with manager or project-leads is "We'll deal with 'that' issue after launch" or "Yeah, we need to hit the deadline, so let's just get this thing working, and deal with the fallout later". Dealing with problems retroactively is always going to be more expensive and some problems simply aren't fixable after a product launch. If you're going into product with known bugs or defects, at least let the technical people choose which bugs.
I see sibling comments on big rewrites. No. I see sibling comments on service based replacements. Still no.
Put structure in. Add tests. Rinse. Repeat.
That's a technical thing you can try, in addition to any of the social approaches the other comments are suggesting.
Using Hack makes the incremental approach more bearable on your people, and bear more fruit than just staying in plain PHP.
Formally: in performance review. Informally: acknowledge their work.
also, find someone who thrives on eliminating crap and let them get stuck in
I really like to eliminate crap, but I also try to avoid doing it as much as possible because it's something that never earns any recognition or praise. Historically I've had to really push to even get my boss to recognize it as an accomplishment during my quarterly reviews. Maybe management says they don't like technical debt, but that doesn't mean anything if incentive structures are designed in ways that encourage me to pile it on ever higher.
"oh we've loads of technical debt, it's such a pain", becomes "well we ticked off 4 big TD milestones last quarter and accrued a new small one"
- Don't bite off more than you can chew. You could "replatform" and try to replace everything that currently works with new tech. In my experience, while the result is an admirable amount of sophisticated technology, the value for the business is unrealizable for a significant amount of time (which usually results in "bad things," like your stock going down, employees being unhappy/leaving due to thinking that it's not going to work out, sales not hitting targets because they don't believe in the product they're selling, customers having more "strength" during contract/sales negotiations, etc). I have observed ~4 years of significant (maybe over a hundred engineers) investment in an effort to replace the entire technology of a relatively young company. I have read many stories of companies doing this and going belly-up as what appears to be a direct result (few companies survive the process; Uber would be an example of a company that is doing this and will survive[1] -- Steve Blank has some particularly appropriate reading material[2]). The problem is the business must continue to grow and sell its product during this time (stable business is important, but growth is critical -- and you can't focus on growth when you're rewriting your technology from the ground up). This seems obvious, but the moment I hear someone say "greenfield" when they also have significant tech debt, I raise an eyebrow of suspicion.
- So, following that, do bite off very small chunks and slowly decompose your tech debt into whatever your well-paid, highly-competent team lead(s) recommend in terms of architecture. For example, if you have a huge legacy application, slowly separate each logical component into its own microservice (assuming your team leads believe microservices are the best architecture for your use-case, etc).
- Freelancers/students/interns/contractors are great! But don't let them design anything. I say this not as a jab to anyone in this category. I work with people in these categories daily. However, it's critical that their work is only implementation and that it is written in a way that has absolute minimal cognitive overhead. The reason is because if you hire someone temporarily to produce a hoozit that does Thing, then in the absolute best case, they will produce exactly that, but you (and your well-paid, highly-competent staff) will have absolutely no idea or understanding as to how or why the hoozit does Thing, or how to modify the hoozit to be a whatsit, or make the hoozit do Otherthing. Sure, you could figure it out. But I posit the cost of doing so is greater than the cost of having done it yourself, even if it takes longer. And now that I think of it, this point is supported strongly by your very own experience already: temp workers always produce technical debt, even in the absolute best case (an example of a worse case is that they produce an unmaintainable/incomprehensible/unmodifiable hoozit that does not do Thing or only does Thing in some conditions a.k.a. being riddled with bugs). Competency or compensation has little or no effect. My belief is that this is often because temp workers know their position is temporary and will strive to achieve exactly Thing when building your hoozit -- they are economically motivated to do so. They are not economically motivated to build your hoozit in such a way that it can become a whatsit later on.
- Remember to deliver value -- in particular, drive growth. This is super important in the technology industry. You should always have some amount of your engineering muscle focused on delivering new value. Sometimes that is reducing technical debt, or decomposing your legacy application, or building new products. Sometimes it's integrating a third-party's product with yours in some way, as "uncool" as that may sound. Sometimes you are in a bind when you can't produce new value without dealing with technical debt. So deal with the technical debt in the most sensible way (i.e., not producing additional technical debt, but also not foregoing any work to reduce your technical debt in the pure interest of improving the top line). If you think of your legacy application as delivering some value, but you don't ever add new features to it, consider your competition and whether your business will lose critically due to lack of innovation.
- I believe that the best engineers care a lot more about understanding the business and how their work aligns with it. If you ask your team, I expect they will tell you that they would rather work on the mind-numbing effort, in some way, if it is the best thing for the business. Junior software developers will always prefer greenfield. More senior software developers will seek the optimal solution. (Similarly to how junior engineers will prefer Shiny New Tech X, whereas senior engineers will only use Shiny New Tech X when it really, really makes sense to do so and there is a strong alignment with the business and/or low risk to doing so -- like in a new product/microservice that is less critical than your core product offerings, for instance.)
- Don't fall into the trap of believing that the only way to grow the business is to abandon/migrate away from legacy applications. At the least, move your users/customers as you decompose/replace legacy software/services. Avoid trying to make big migrations, especially wholesale. (OK, maybe I'm repeating my first point here...)
Separately, I'm curious why the CEO having written any of the code matters if he is no longer contributing code (I'm assuming he is not, since you mentioned he stopped learning new techniques). Or did you mean that he stopped having free time to learn new techniques and therefore stopped contributing code? I would think the CEO needs to stop contributing code as soon as you have enough engineers to meet your minimum desired sprint points (or whatever metric you use for productivity). I certainly don't think that not having learned new techniques necessitates the cessation of contribution, although learning new techniques is a likely byproduct of contribution (due to experience, research, implementation itself, code reads, etc). But, I'm digressing from the topic.
Anyway, hopefully my commentary/experience is helpful to you. Best of luck!
[2] https://steveblank.com/2011/01/25/startup-suicide-%E2%80%93-...
About 4 million lines of PHP code, written by underpaid, sometimes not well meaning, freelancers and students over the span of 8 years. The CEO wrote a large part, but stopped learning new techniques around 2004.
I'm bringing competent and well paid people in through my network and try my very best to give them as much freedom as possible, I allow and encourage greenfield modules/services that run on separate and new infrastructure for anything that is possible to be rewritten in the timeframes, but the larger part of the job is still mind numbing to my team and that makes me question my wisdom.
If anyone has tips on how to steer such a ship in a direction where the work is less frustrating for my devs I'm very open for advice.