You're making a silent assumption that the original team is well informed about why the things are like they are and that they know what they are doing. I think it is not always the case.

I've been to a project once where the mess in the original system was the result of the original team not knowing what they were doing and just doing permutation based programming - applying random changes until it kinda worked. The situation was very similar to that described by the OP. They even chose J2EE just because the CTO heard other companies were using it, despite not having a single engineer knowing J2EE. Overall after a year of development the original system barely even worked (it required manual intervention a few times per day to keep running!), and even an imperfect rewrite done by a student was already better after 2 weeks of coding.

So I believe the level you're starting the rewrite from is quite an important factor.

Then of course there is a whole world of a difference between "They don't know what they are doing" vs "I don't like their tech stack and want to master <insert a shiny new toy here>". There former can be recognized objectively by:

- very high amount of broken functionality

- abysmal pace at which new features are added

The original team may not have been the best at the task, but they still managed to deliver 100 MM in revenue. Sometimes the things they leave behind/ignore simply don’t matter to the business/useful tech.

Particular to ad tech, the lifespan of any particular software is lower than you’d expect (unless your google/Facebook). Technology that pays out big one year will become pretty meh within 3 years. In the case above I’d argue that the new tech team didn’t really understand this dynamic and so they focused on the wrong things such as rewriting functionality that didn’t matter for the future. Or making big bets on aspects of the product which were irrelevant.

To the OP, we don’t know that the lifespan of any of these php files is greater than an individual contract. If the business can be modeled as solve a contract by putting a php file on prod - rewriting may be entirely worthless as the code can be “write once, read never”.

Revenue is a crazy kpi for technical excellence. You should never let a high revenue rely on extremely bad code.

We (my good friend and I who both have 20+ years of experience) were brought in specifically do to the rewrite. We were new to the company. We actually had to rebuild the entire IT department while we were at it as well.

> new tech team immediately committed to rewriting the code base into ruby micro-services

well... sigh.

> These refactors inevitably failed to produce a meaningful impact to revenue

It sounds like less about the refactor itself and more about the skills of the team doing the refactor. You certainly can't expect a refactor to go well if the team makes poor decisions to begin with.

> We were brought in specifically do to the rewrite.

That's the key difference. The stakeholders should always be in on the rewrite.

> It sounds like less about the refactor itself and more about the skills of the team doing the refactor. You certainly can't expect a refactor to go well if the team makes poor decisions to begin with.

This has been my biggest struggle with rewrites where I’m currently working. We have several large, messy old codebases that everyone agrees “needs a rewrite” to (1) correct for all the early assumptions in business needs that turned out wrong, (2) deal with old PHP code that is very prone to breakage with every major new PHP version released, and (3) add much needed architectural patterns specific to our needs.

I’ve seen rewrites of portions of the project work when they involve myself and one other mid-level dev who has a grasp on solid sw engineering practices, but when the rest of the (more senior) team get involved on the bigger “full rewrite”, they end up quickly making all the same mistakes that led to the previous project being the mess that it is.

Sure, it will be using fancy new PHP 8 features, and our Laravel framework will force some level of dependency injection, but the you start seeing giant God classes being injected over here, but duplicated code copy-pasted over there, all done by “senior” devs you feel you can’t question too strongly.

To that end, an open and collaborative culture in which you start the rewrite with some agreed upon principles, group code reviews and egos kept in check, are all necessary for this to work.

You have a great experience and did a great job indeed. My only question is how does one get 20 years of such experience without horrific flashbacks of “let’s just rewrite it” decisions. Do you do rewrites/redesigns often? What’s your success rate?

I've done what I would consider as four rewrites that I can remember as large events in my life (although not fully what you'd expect). But all are good stories in my opinion.

First one was the above example. It was for the largest hardcore porn company on the planet. Myself and my good friend Jeff rebuilt an already successful business IT department from the ground up and made it even more successful. Ever heard of 'the armory in sf'?

Second was that jeff and I were hired as contractors by Rob @ Pivotal Labs (ceo) to help the CloudFoundry team rewrite itself after he had bought the team and trimmed it down to only the good people. That one was a huge mess. We spent a lot of time deep in shitty ruby code using print statements trying to figure out what 'type' of an object something was and, of course, backfilling tests. It was a fun project and both Jeff and I learned the Pivotal way, which was probably the most enlightening thing I had ever learned about how to develop software correctly from a PM perspective. If you want to improve your skills beyond just slinging code, spend some time figuring their methodology out. Much of it is documented in Pivotal Tracker help documentation and blog posts.

Third one was not really a rewrite, but the original two founders, who were not technical, had tried to hire a guy and got burned because the guy couldn't finish the job. Sadly, they had already paid the person a ton of money and got really nothing functional out of it. We (jeff and I again!) just started over. We did a MVP in 3 months (to the exact date, because we both know how to write stories using pivotal tracker and do proper estimates) and ended up doing $80m in revenue, in our first year with an initial team of 5 people.

Fourth one was three guys (who were also not technical) I kind of randomly met after I moved to Vietnam. They were deploying litecoin asic miners into a giant Vietnamese military telco (technically, they are all military). They had hired another guy to do the IT work and he was messing it all up. They invited me out to help install machines, I came out, rebuilt their networking layout and then proceeded to eventually fix their machines because the software 'firmware' that was on them was horrible. I also added monitoring with prometheus so that I could 'see' issues with all these machines. That first day on the job, they fired the other guy and made me CTO. We ended up deploying in another datacenter as well. It was a really wild experience with a ton more stories.

Life has been, um, interesting. Thanks for reading this far.

Please tell me that you've retired now due to your incredible billing rates and track record of success.

Not everything has been a success. For example, unless you're stupid rich and can afford years of losses, never start/own a night club or you might end up working for the rest of your life to pay off your debts.

The problem is that most developers are crap and self centered on working with the tech they like.

You need to work with someone who doesn't care about filling up their CV with "ruby microservices" and get stuff done.

If I went into a business to do a rewrite and decided to use $shinyNewTech because I want to build up rust experience I'd probably end up wasting years with little results.

The existing app was a large rails monolith. This wasn’t a small 10 person team but a 50 person org. Groups can get funny ideas sometimes.

> why does our tracking pixel return a purple image?

Now I'm really curious, is there some exciting non-obvious reason for a tracking pixel to be purple? Was it #FF00FF or more like #6600DD?

This definitely needs an answer.

In fact, until OP can give us the right answer, we immediately need even wrong answers!

You reading this. Yes, you. Give your best wrong answer below.

My best wrong answer is that there were different colored pixels for different front-end versions, and the app had some radically different responses depending on the version. Maybe MENA would return white, SE Asia green, people who signed up during a sale would return blue, whatever. After a while, the other pixels were removed and only one shade of purple were used for everyone, but the code for processing them was not removed. So now, if the tracking pixel is not a precise shade of purple, some unexpected shenanigans ensue.

I worked on an app once that used two different, equally ancient libraries to a) generate thumbnails and b) create a png from a pdf. While modifying part of this process I started realizing that there were conditions where you'd get a PDF thumbnail at the end, but its output had a red tint to it.

Input looked fine and invoking each step manually worked fine as well.

Come to find out that certain PDFs contained color calibration information that, combined with how we were calling it, would treat ARGB as RGB. The input would have transparency info defined and the thumbnail generator would happily repurpose the alpha channel as the red channel instead.

The tracking pixel was made my scaling the company logo down to a 1x1 image.

That's brilliant! That way nobody could accuse you of spying. "It's just our logo. What's all the fuss about?"

Obviously it's because mauve has the most RAM.

Page background where pixel displayed was purple

Accessibility. Protanopia affects cones perceiving red color.

Obviously !! The anti-doppler shift trick /s :)

I did a rewrite of a 30 year old bit of perl/php2 over the last year. Not knowing why things were the way they were was really useful for the younger team members and me to get familiar with the codebase and the business context.

Anecdotal: I asked people why they keep incorrectly using jQuery methods and produce ambiguous, difficult-to-maintain code in the year of 2022. (we still have jQuery as a dependency for legacy code.) The response was that they were not aware that native counterparts like document.querySelectorAll exist in the browser. They just copied the old jQuery code, modified them, and it worked.

I am pretty sure this kind of thing exists in any large legacy codebase.