As an example, I worked at an ad-tech startup that swapped it's tech team out when it had ~100 million in revenue (via acqui-hire shenanigans). The new tech team immediately committed to rewriting the code base into ruby micro-services and were struck by strange old tech decisions like "why does our tracking pixel return a purple image?". The team went so far as to stop anyone from committing to the main service for several years in a vain attempt to speed up the rewrite/architecture migration.
These refactors inevitably failed to produce a meaningful impact to revenue, as a matter of fact the company's revenue had begun to decline. The company eventually did another house cleaning on the tech team and had some minor future successes - but this whole adventure effectively cost their entire Series D round along with 3 years of product development.
I've been to a project once where the mess in the original system was the result of the original team not knowing what they were doing and just doing permutation based programming - applying random changes until it kinda worked. The situation was very similar to that described by the OP. They even chose J2EE just because the CTO heard other companies were using it, despite not having a single engineer knowing J2EE. Overall after a year of development the original system barely even worked (it required manual intervention a few times per day to keep running!), and even an imperfect rewrite done by a student was already better after 2 weeks of coding.
So I believe the level you're starting the rewrite from is quite an important factor.
Then of course there is a whole world of a difference between "They don't know what they are doing" vs "I don't like their tech stack and want to master <insert a shiny new toy here>". There former can be recognized objectively by:
- very high amount of broken functionality
- abysmal pace at which new features are added
Particular to ad tech, the lifespan of any particular software is lower than you’d expect (unless your google/Facebook). Technology that pays out big one year will become pretty meh within 3 years. In the case above I’d argue that the new tech team didn’t really understand this dynamic and so they focused on the wrong things such as rewriting functionality that didn’t matter for the future. Or making big bets on aspects of the product which were irrelevant.
To the OP, we don’t know that the lifespan of any of these php files is greater than an individual contract. If the business can be modeled as solve a contract by putting a php file on prod - rewriting may be entirely worthless as the code can be “write once, read never”.
> new tech team immediately committed to rewriting the code base into ruby micro-services
well... sigh.
> These refactors inevitably failed to produce a meaningful impact to revenue
It sounds like less about the refactor itself and more about the skills of the team doing the refactor. You certainly can't expect a refactor to go well if the team makes poor decisions to begin with.
That's the key difference. The stakeholders should always be in on the rewrite.
This has been my biggest struggle with rewrites where I’m currently working. We have several large, messy old codebases that everyone agrees “needs a rewrite” to (1) correct for all the early assumptions in business needs that turned out wrong, (2) deal with old PHP code that is very prone to breakage with every major new PHP version released, and (3) add much needed architectural patterns specific to our needs.
I’ve seen rewrites of portions of the project work when they involve myself and one other mid-level dev who has a grasp on solid sw engineering practices, but when the rest of the (more senior) team get involved on the bigger “full rewrite”, they end up quickly making all the same mistakes that led to the previous project being the mess that it is.
Sure, it will be using fancy new PHP 8 features, and our Laravel framework will force some level of dependency injection, but the you start seeing giant God classes being injected over here, but duplicated code copy-pasted over there, all done by “senior” devs you feel you can’t question too strongly.
To that end, an open and collaborative culture in which you start the rewrite with some agreed upon principles, group code reviews and egos kept in check, are all necessary for this to work.
First one was the above example. It was for the largest hardcore porn company on the planet. Myself and my good friend Jeff rebuilt an already successful business IT department from the ground up and made it even more successful. Ever heard of 'the armory in sf'?
Second was that jeff and I were hired as contractors by Rob @ Pivotal Labs (ceo) to help the CloudFoundry team rewrite itself after he had bought the team and trimmed it down to only the good people. That one was a huge mess. We spent a lot of time deep in shitty ruby code using print statements trying to figure out what 'type' of an object something was and, of course, backfilling tests. It was a fun project and both Jeff and I learned the Pivotal way, which was probably the most enlightening thing I had ever learned about how to develop software correctly from a PM perspective. If you want to improve your skills beyond just slinging code, spend some time figuring their methodology out. Much of it is documented in Pivotal Tracker help documentation and blog posts.
Third one was not really a rewrite, but the original two founders, who were not technical, had tried to hire a guy and got burned because the guy couldn't finish the job. Sadly, they had already paid the person a ton of money and got really nothing functional out of it. We (jeff and I again!) just started over. We did a MVP in 3 months (to the exact date, because we both know how to write stories using pivotal tracker and do proper estimates) and ended up doing $80m in revenue, in our first year with an initial team of 5 people.
Fourth one was three guys (who were also not technical) I kind of randomly met after I moved to Vietnam. They were deploying litecoin asic miners into a giant Vietnamese military telco (technically, they are all military). They had hired another guy to do the IT work and he was messing it all up. They invited me out to help install machines, I came out, rebuilt their networking layout and then proceeded to eventually fix their machines because the software 'firmware' that was on them was horrible. I also added monitoring with prometheus so that I could 'see' issues with all these machines. That first day on the job, they fired the other guy and made me CTO. We ended up deploying in another datacenter as well. It was a really wild experience with a ton more stories.
Life has been, um, interesting. Thanks for reading this far.
You need to work with someone who doesn't care about filling up their CV with "ruby microservices" and get stuff done.
If I went into a business to do a rewrite and decided to use $shinyNewTech because I want to build up rust experience I'd probably end up wasting years with little results.
Now I'm really curious, is there some exciting non-obvious reason for a tracking pixel to be purple? Was it #FF00FF or more like #6600DD?
In fact, until OP can give us the right answer, we immediately need even wrong answers!
You reading this. Yes, you. Give your best wrong answer below.
Input looked fine and invoking each step manually worked fine as well.
Come to find out that certain PDFs contained color calibration information that, combined with how we were calling it, would treat ARGB as RGB. The input would have transparency info defined and the thumbnail generator would happily repurpose the alpha channel as the red channel instead.
I am pretty sure this kind of thing exists in any large legacy codebase.
Figure out the single most important flow in the application - user registration and checkout in an e-commerce app, for example.
Write an automated end-to-end test for that. You could go with full browser automation using something like Playwright, or you could use code that exercises HTTP endpoints without browser automation. Either is fine.
Get those running in GitHub Actions (after setting up the git scraping trick I described here: https://news.ycombinator.com/item?id=32884305 )
The value provided here immense. You now have an early warning system for if someone breaks the flow that makes the money!
You also now have the beginnings of a larger test suite. Adding tests to an existing test suite is massively easier then starting a new test suite from scratch.
Let's say you start to write tests and start to see issues crop up. Now what? How do you fix those things?
Github actions!? They don't even have source control to begin with. There are so many steps necessary to just get to that point, why bother?
If the existing code base already has extremely slow movement and people are unwilling to touch anything for fear of breaking it... you're never going to get past that. Let's say you do even fix that one thing... how do you know it isn't breaking something else?
It is a rats nest of compounding issues and all you are doing is putting a bandaid on a gushing open wound. Time to bring in a couple talented developers and start over. Define the MVP that does what they've learned their customers actually need from their 'v1' and go from there. Focus on adding features (with tests) instead of trying to repair a car that doesn't pass the smog test.
I assumed the tests wouldn't be for correctness, but for compatibility. If issues crop up, you reproduce the issues exactly in the rewrite until you can prove no one depends on them (Chesterton's fence and all).
The backwards-compatibility-at-all-costs approach makes sense if the product has downstream integrations that depend on the current interface. If your product is self-contained, then you're free to take the clean slate approach.
You're assuming that the people coming in to write these tests can even make that distinction. How do you even know what the compatibility should be without really diving deep into the code itself? Given how screwed up the codebase already is, it could be multiple layers of things work against each other. OP mentioned multiple versions of jquery on the same page as an example.
Writing tests for something like that is really a waste of time. Better to just figure out what's correct and rewrite correct code. Then write tests for that correct code... that's what moves things forward.
You can pretty much black-box the code and only deep dive when there are differences. Here's what I've done in the past for a rewrite of an over-the-network service:
1. Grab happy-path results from prod (wireshark pcap, HTTP Archive, etc), write end-to-end tests based on these to enable development-time tests that would catch the most blatant of regressions.
2. Add a lot of logging to the old system and in corresponding places in the new system. If you have sufficient volumes, you can compare statistical anomalies between the 2 systems
3. Get a production traffic from a port mirror, compare the response of your rewritten service against the old service one route at a time. Log any discrepancies and fix them before going live, this is how you catch hard-to-test compat issues
4. Optionally perform phased roll out, with option to roll back
5. Monitor roll out for an acceptable period, if successful, delete old code/route and move to the next one.
The above makes sense when backwards compatibility is absolutely necessary. however, the upsides is once you've set up the tooling and the processes, subsequent changes are faster.
We're talking about a webapp here, not rocket science.
When you keep finding bugs like that while refactoring and making things better, it will demoralise you. The productivity will stop when that happens.
It also require above average engineers to fix the mess and own it for which there is not much benefit.
Your refactoring broke things? Now it's your turn to fix it and also ship your deliverables which you were originally hired for. Get paged for things that weren't your problem.
If I was a manager and assigned this kind of refactoring work, I will attach a significant bonus otherwise I know my engineers will start thinking of switching to other places unless we pay big tech salaries.
People keep quoting Joel's post about why refactoring is better than rewrite but if your refactor is essentially a rewrite and your team is small or inexperienced - it's not clear which is better.
Parallel construction and slowly replacing things is a lot of unpaid work. Just the sheer complexity of doing it bit by bit for each piece is untenable for a 3 person team where most likely other two might not want to get into it.
That's not true, it doesn't require above average engineers. It requires a tech lead that has the desire and backing to make a change, and engineers willing to listen and change. It doesn't require a 10x engineer to start using version control, or to tell their team to start using version control for example..
It is a can of worms.
Been there, done that. Slightly differently where they had a test server and prod server. So already better except one day I made a change and copied to prod. Yes it was manual. Just scp the files over to prod. And stuff broke. Turned out someone had fixed a bug directly in prod but never made the change on the test server.
First thing I did was to introduce version control and create a script to do make deployment automatic meaning it was just a version control update on Prod (also scripting languages here). Magically we never had an issue with bugs reappearing after that.
Pretty simple change and you can go from there.
The above code base was over 20 years old and made use of various different scripting languages and technologies including some of the business logic being in stored procedures. Zero test coverage anywhere. You just 'hide' small incremental changes to make things better in everything you do. Gotta touch this part because they want a change? Well it could break anyhow so make it better and if it breaks, it breaks and you fix it. It needs judgment though. Don't rewrite an entire module when the ask was adding a field somewhere. Make it proportional to the change you need to make and sometimes it's not going to be worth it to make something better. Just leave it.
Not rational but folks don't have to explain their feelings. You will be hated.
Now as for how to get the other devs on board, I agree with you that you can't just barge in and tell them everything they are doing is wrong etc. I never said to do that and I'm replying to a specific comment in the thread not the original Ask HN.
I.e. when I write about what I've done in the past, I got buyin from my boss and my colleagues on what I was going to do. But I didn't just sit there and kept doing what they had done over the past years. I changed lots of other little things too in the same manner.
So if we do want to talk about the original Ask HN and how to get the existing employees not to hate you, you can start by letting them tell you about what they think the problems are. What are their pain points. They might just not know what to do about them but actually see them as problems too. Maybe they've tried things already but failed or got shot down by others in the company. Maybe they did try to introduce version control but their non tech boss shot them down.
Of course it may not work out. Some people really are just stupid and won't listen even if you try to help them and make them part of the solution.
None of what I said is a big-company process in any way. If in your book using source control is a big company process that will sink a startup then be my guest and I will just hope we never have to work together. Source control is a no-brainer that I even use just for myself, have used in teams of two and teams dozens to hundres. The amount of process around is what scale with the kind of company. Source control is useful by itself in every single size of company.
Code review, coding standards, required tests for everything, multiple stages of deployment - are not simple and can stall development. Done wrong they can sink a company.
It's easy to read the worst possible construction on what other people write here. It's never a good idea.
Btw I worked at a startup for 8 years. It was still a startup, depending on new investment to meet the monthly. In any case the described dev group was behaving in a way that used to be typical of startups. And even business units in larger organizations have runway.
How do they test things on production? If there’s a bug how do they revert to the previous version? There are way more issues without source control than with.
Right: no point in adding any tests until you've got source control in place. Hence my suggestion for a shortcut to doing that here: https://news.ycombinator.com/item?id=32884305
You're trying to spec features on a moving target.
Even if they were able to do 50% time on the rewrite you'll never actually get to feature parity.
The only viable plan, unless the company has an appetite to triple the dev headcount, is to set an expectation that features will have an increased dev time, then as you spec new features you also spec out how far you will go into the codebase refactoring what the new features touch.
What we did was make the case that we could increase revenue by being able to add valuable features more easily/quickly. We started with a super MVP rewrite that kept the basic valuable features, launched, then spent the rest of our time adding features (with tests). Hugely successful.
The key, of course, will be to get 1-2 top notch developers in place to set things up correctly from the beginning. You're never going to be effective with a few jr's who don't have that level of experience.
It's $20m functional. It's possible it could be better but unless this is the kind of huge org where 20m is nothing (doesn't sound like it) you really need the behaviors documented before you start screwing with it. It's very likely this thing has some pretty complex business logic that is absolutely critical to maintain.
Nothing I said suggested otherwise. Absolutely critical for whomever is doing a rewrite to understand everything they can about the application and the business, before writing a single line of code.
Many of us have been in this exact position before, multiple times. Many of us have seen somebody say "our only choice is a full rewrite" - some of us were the one making that decision. Many of us have seen that decision go disastrously wrong.
For me, the problem was my inability to do what I'm good at: write tests, write implementations that pass that test, etc. Every time I suggested doing something, somebody would have a reason why that would fail because of some unclear piece of the code. So rather than continuously getting blocked, I tried to step into my comfort zone of writing greenfield code. I built a working application that was a much nicer codebase, but it didn't match the original "spec" from customer expectations, so I spent months trying to adjust to that. I basically gave up managing the team because I was so busy writing the code. In the end, I left and the company threw away the rewritten code. They're still in business using the shitty old codebase, with the same development team working on it.
If you really want to do the rewrite, accept how massively risky and stressful it will be. The existing team will spend the whole team trying to prove you were wrong and they were right, so you need to get them to buy into that decision. You need to upskill them in order to apply the patterns you want. And you need to tease apart those bits of the codebase which are genuinely awful from those that for you are merely unfamiliar.
Personally, I would suggest a course for you like https://www.jbrains.ca/training/course/surviving-legacy-code, which gives you a wider range of patterns to apply to this problem.
There is a lot of evidence rewrites are hard to do well, and especially prone to failure.
…you might pull it off, it’s not impossible, sure. …but are you seriously saying it’s the approach everyone should take because it worked for you once?
Here my $0.02 meaningless anecdotal evidence: I’ve done a rewrite twice and it was a disaster once and went fine the second time. 50% strike rate, for me, personally, on a team of 8.
What’s your rate? How big was your team, how big was the project? What was the budget? Did you do it on time and on budget? It’s pretty easy to say, oh yeah, I rewrote some piece of crap that was a few hundred lines in my spare time.
…but the OP is dealing with a poorly documented system that’s very big, and very important and basically working fine. You’re dishing out bad advice here, because you happened to get lucky once.
Poor form.
Good advice: play it safe, use boring technology and migrate things piece by piece.
Big, high risk high reward plays are for things you do when a) things are on fire, or b) the cost of failure is very low, or c) you’re prepared to walk away and get a new job when they don’t work out.
Uhm. The tests don’t do any such things.
> It is better to spend the time figuring out what all the features are, document them
Yes. And the tests you should write are executable documentation showing how things are. It is like taking a plaster cast of a fossil. You don’t go “i think this is how a brachiosaurus fibula should look like” and then try to force the bones into that shape. You mould the plaster cast (your tests) to the shape of the fossil (the code running in production). Then if during excavation (the rewrite) something changes or get jostled you will know immediately that it happened, because the cast (the tests) no longer fit.
Which sure beats some other company coming along and "rewriting" the same or similar functionality in a competing product and killing your own revenue. But it does come down to how big the codebase is and how long it would take for an MVP to be a realistic replacement. If there are parts that are complex but unlikely to need changing soon you can usually find ways to hide them behind some extra layer. Is there any reason you couldn't just introduce proper processes (source control, PRs, CI/CD etc.) around the existing code though?
I'll also say there's a lot of semantics at play here. What is a "rewrite", what is a "test" vs a "document", what is "functional"? I read your main point being that one should avoid sunk-cost fallacy and find the right places to cut bait and write off unsalvageable pieces. The art of major tech debt cleanup is how big of pieces can you bite off without overwhelming the team or breaking the product.
This is not TDD; it's writing tests to confirm the features that work now. Then, when you make changes, you can get an early warning if something starts going south.
The blogs are plainly stating, "even though you feel you should rewrite, you probably shouldn't."
What is really needed (and almost definitely doesn’t exist) is some kind of spec for the software.
I gotta love hacker news, people who think the fact a backend is written in horrid PHP means it is "already not functional" while they spend their days learning something like Haskell that make them negative revenue per year.
I also don't know Haskell and have no desire to learn it. I prefer to build products in static compiled languages where I can more easily hire developers.
That's simply not true. I've inherited something just as bad as this. We did a full rewrite and it was quite successful and the company went on to triple the revenue.
> get some testing in place
Writing tests for something that is already not functional, will be a waste of time. How do you fix the things that the test prove are broken? It is better to spend the time figuring out what all the features are, document them and then rewrite, with tests.