Perhaps their metrics tell them that all of this is fine, I don’t know. But then, you have to realize Google has so much money they don’t really have to spend it very efficiently. For everyone else, this approach to rewrites seems like an extremely expensive way to produce software that’s not even all that great.

Perhaps their metrics tell them that all of this is fine, I don’t know. But then, you have to realize Google has so much money they don’t really have to spend it very efficiently.

The user experience ranges between good to mediocre to bad, depending. Google is simply too rich and powerful to care. So long as the goose keeps laying the golden eggs, they can just keep going along and patting themselves on the back.

But some one got to tick some boxes on there promotion track

Bug fixes and incremental features will generally not get you promoted for good reasons, we expect senior engineers to have system design skills and there is simply no way to demonstrate that without having your engineers design systems. If you only assign tasks based on the needs of your product and not on the needs of your workforce you could easily find yourself with a critical skill shortage.

There's a certain inefficiency to this, but only if you put your blinders on. If you have engineers churning out meaningless work then you certainly need to address that problem, but if you prioritize short-term product success over team health you are only trading one problem for another.

This is exceptionally bad, but sadly true.

If you have an engineer who can unblock teams and fix issues in an hour that others take a week or cannot fix at all, they are gonna jump ship if they can't be recognized.

At that point you've lost a valuable resource.

If I can do that without it being a fluke, they’d better bump my salary.

Even if the product/library/framework never changed much, rewrites would still be necessary to keep it going as new generations of programmers shuffle through. Otherwise we wind up with a Verner Vinge dystopian culture of software archeology.

An added kicker, I have actually gone down this road, and later realized the only thing I needed to do was make a small change to the existing system, so I did just that and discarded the new work.

Business got their new features, delivered on time, the system was still stable and reliable, everyone was happy, and now my domain knowledge has increased significantly so that new features and development or bug fixes will be even faster.

Out of date dependencies can be seen as code rot, regardless of quality/state of the actual code.

(Not saying that it should be rewritten because of that, but I do consider outdated dependencies to be code rot.)

Trying to make the software "theirs" seems to be an issue at Google, at least with their open source software and has seemed to have lead to it being less reliable.

For example, Angular Material v1 was one of the most complete and stable front-end packages on the market a few years ago. Then, back in 2017 the lead developer for the project was replaced with a new dev.

This new dev then went about assigning every issue and pull request to himself, modifying or rejecting pr's that had previously been approved, closing issues that had in progress pr's as won't fix, locking discussions, and just generally breaking stuff. (I've personally had to peg my project to v1.4 because everything since 2016 has been a regression)

If you go to the Angular Material v1.x Github page today, you'll see the same dev on pretty much everything.

This isn't productive ownership as it prioritizes the engineer over the customer and has lead to a generally broken system from one of the most stable properties out there. Not to mention, these open-source projects are most people's first exposure to Google's code... Having them be unpredictable regarding the functionality of their software with little concern for users/contributors in the name of making their devs feel special seems like a bad model to learn from.

I also don’t think it’s laziness. Developer hubris is real and it can get in the way of actual business value.

You pretty much nailed everything else. I share the sentiment because I’m currently working on splitting up a monolith into microservices. Most of the “quick wins” are things that didn’t need to be rewritten anyway, and the stuff that actually sucks is difficult to address. It’s tricky to get right.

Nobody said it is good general advice. There is so much stuff you need to do well to have a good chance at successful rewrite that it become good advice that it is a bad idea. Nobody said rewrites cannot be done. If you have a company that knows how to do rewrite (and makes it constantly which, I guess, helps a lot) then a lot of problems can be solved by starting from scratch and it it may be it is well worth it.

A sense of ownership and responsibility over a codebase is fundamental and essential to proper stewardship and maintenance of that code, and refactoring and rewriting is the most effective way to inculcate that feeling. Sometimes it’s not always feasible, and sometimes it’s not necessary, but the end state is essential. Group or shared ownership of code is a manager’s wet dream but pragmatically impossible, a swamp of mediocrity.

What is the current programming language used?

Is the project mostly legacy (few or no changes in the last few years)?

Is the project critical to the company?

How many people are working on it today? Are they fully assigned to it or is it a "touch when it breaks" kind of situation?

Is the project following modern CI/CD practices? If not, how hard would it be to adapt in the current software stack? Would it be easier in a new stack?

Are developers spending more time than expected understanding the code base?

Is it a political situation? Is the current owner refusing changes? Could it be that rewriting it is just a costly way for removing ownership from that developer/manager?

The fact that "write-only code" is apparently considered a part of sensible software-engineering practice speaks volumes about what their technical culture is like more generally. One would think that code should be much easier to read and survey than it was to write.

In fact, I'll often sketch out a block diagram or some pseudocode if I'm having trouble grokking something I'm reading just to help me get into the proper mindset. I agree this is a problem with the current state of the field, but I haven't seen any good solutions, only hacks and workarounds.

If the intention is for a uniquely skilled human to give a computer instructions, and the computer to execute them - then just write it in machine code.

We need to be precise about what the problem is, exactly. Here is a more salient formulation: "Not Invented Here" is a form of bias that distorts cost/benefit analysis when people are deciding whether to reuse existing code.

They get ticked off by one imperfection here and other over there and immediately run for exit shouting “I could do so much better”.

Given the formulation of NIH as a form of bias that distorts cost/benefit analysis, what can we do at this point? How about quantifying the cost of the "one imperfection here and other over there" instead of going by the subjective being "ticked off?"

Instead of understanding why things are the way it is, they fantasize about how they can one up original authors and claim their own hero title. They go on to throughly underestimate the time to recreate what has taken years of learning.

To complete the cost/benefit analysis, we can then quantify the development cost of the library they are considering re-implementing.

Meanwhile competition has moved on to V2 laughing their way to the bank and customers scratch their heads why you are still stuck in same place for so long. Then our new “owners” gets their promos after massive marketing of how much better everything is now.

This needs to be quantified and put into the cost/benefit analysis.

But to everyone’s surprise they soon leave the project because working on bugs and incremental features has became boring and BTW, the new stuff is just as complex as old stuff. New devs roll in and we start the whole cycle again.

This also needs to be quantified and put into the cost/benefit analysis. Instead of doing cost/benefit analysis attached to individuals, it should be done by product or by project, so that these turnover costs are also accounted for.

It's perfectly reasonable (for the reasons in the submission) to rewrite code that exists already.

Hell, if they really care about claiming that hero title, why not spend that effort to understand and refactor the parts of the code that were initially hard to deal with. What you wrote above is a distillation of what a "tech bro", "Type-A player", ego-based culture looks like. It is the opposite of true professionalism.

A system such as Gmail is composed of many smaller parts. If one of those parts was written years prior for a world that has since changed, it may be accruing technical debt as it's continually extended to fit new requirements. An occasional rewrite helps address this type of decay.

Without the rewrite you may find yourself 10 years later with a system that's both critical and kludgy, and at that point the rewrite will be a much larger project.

I like how you state this like it's an objective fact. I've always been happy with the gmail UI and the latest iteration is great too. Outlook on the other hand...

Be too quick and you're constantly chasing shadows… wait too long and you're immobile.

To get on-topic: Gensim could use such a rewrite as well… The ML world changed, expectations and requirements changed, ecosystem and APIs changed. I changed too.

Apparently me spending half a day reading code is no big deal but cleaning it up is a waste of time.

Shrug.

I find this is particularly common mistake by junior programmers (not saying this applies to you), presumably because they aren't used to reading other people's code. Frustratingly, this is often coupled with an attitude that missing out large chunks of existing user functionality is acceptable if it makes the code a bit simpler.

Of course, sometimes rewrites/refactors really are an improvement. Sometimes code really is fragile and confusing, either because of who wrote it or because it has had many small changes tacked on in the easiest places. Or perhaps the last person that understood that code has left the company, so it's OK that you find the code clearer only because you just wrote it! But in any case, it is fair to ask for real justification for a rewrite.

You can say it disrupts others understanding of shitty old code, frankly I don't care. Code is not immutable and maintaining status quo helps nobody except the old guard stay relevant.

> Of course, sometimes rewrites/refactors really are an improvement

If that's what you were talking about in the first place, fair enough and my apologies.

Those perfect one line fixes are great but they approach their limits and eventually need to be refactored. This is natural and should not be frowned upon. Likewise you can't just refactor all the time, one line fixes are faster and less costly in all sorts of ways. The key is to pick and choose when to use each strategy.

I think we're too often looking for simple rules. There are none. Just a bunch of guidelines.

I'll try to give an example I hope is realistic:

One of the heaviest things an application can get is a complete theme system. Suppose you don't have one. It's not a requirement.

Adding a theme system when there is none is months of work and might impact basically every line of code that displays anything.

So you're not doing it. Now for some exceptional system there is just one case where somewhere you are displaying something under an external widget that doesn't meet its size constraints or whatever - long story short your text is invisible, you want to inverse the font to get it working. You don't have any code like that.

Do you think it is OK to add it to the easiest possible place: in this case perhaps you add an optional argument called "need_to_invert_color" (this awkward phrasing tells you it's a hack) to a single function, default it as false, comment it as: //invert the color of the font. This is needed where an external graphing widget with a black background leaks onto our canvas due to not respecting our pixel boundaries, so that our text displays over it.

And then where you call comment the same thing, that //currently a bug in the widget code makes the widget leak xyz pixels below its bottom border. As a temporary fix we introduce an argument need_to_invert_color into our display function. As of this writing 3 Jan 2019 we are just using it from here. The correct fix would be for the widget to stop leaking instead, and when that is done white text is unnecessary - and we might not notice. So we start by testing whether the area we will be overlaid over is indeed the wrong color.

ETC. In other words a quick hack for a corner case, that doesn't fix the underlying bug (workaround) and even as a hack makes use of something that doesn't exist (a theme system), instead adding and documenting a half-assed thing tacked on.

When exactly is the right time can be hard to figure out. Especially because, if enough hacks have accumulated that code really does need a reshuffle, then the job of refactoring is harder, which actually increases the temptation to put it off. But I wouldn't make a big sweeping change just because of one small hack that's a bit ugly.

This was my thinking on a lot of projects. But you know what? That rewrite never became necessary!

So not only did I not start with the "right" architecture - I didn't end with it either!

That's what "many small hacks in the easiest of places" reminded me of and I wondered if you in fact do approve of it. It has always seemed fine for me. Just no problem at all. But with documentation right there and the worse the hack, the clearer the documentation right there explaining and justifying it, up to and including "I don't know why this works but this system call makes the next line succeed, whereas removing it causes the next line to fail sometimes - this is tested in testxzy." Obviously a hack, a terrible hack if you don't know why it works. And a project can end up with a lot of these.

You refactor it.

Now one person understands the system.

Now it takes 2 hours.

I’ll take that.

Sometimes you gotta sit down and bite the bullet and read and ask questions.

I think Google partially does this in order to keep its engineers happy, as you are more happy when you develop something from the ground-up compared to just maintaining something already built. By going down this route those engineers are kept in a “happy state” so there’s less risk of them flying off to other pastures, where they could potentially build the next product that could “kill” Google. Sort of invisible golden hand-cuffs, if you will. I personally find it tremendously wasteful at a societal level but I can see the value of this strategy for Google as a company.

I mean, I guess the duplicate code/class is now coupled to the two places that use it, but I have a hard time seeing how that is worse than two duplicate instances of the code.

Two pieces of code in different parts of the codebase are very similar. They have nothing to do with each other - even semantically. But the code is very similar. So someone thinks this is code duplication and creates a function/class/whatever that both pieces of code can use. Repeat all over the place.

Then one day, one of those two places needs custom behavior. I can either change that function/class and create complexity (have to now support two use cases). Or I can stop using that function/class in that place and go back to the old solution. Sometimes, this is quite a lot of work as aggressive "DRY" leads to a fair amount of coupling - there could be a few layers of DRY'd code there to untangle.

I put "DRY" in quotes because none of this really is DRY. DRY originally was about requirements - not code. No requirement should show up in multiple places in the code base. In this example, even though the code was almost identical in both places, there was little else common. They dealt with different requirements, for completely different reasons. They should never have been refactored to use a common function/class.

These days people keep talking about over-use of DRY, but they're really complaining about overabstraction of disparate code - not the DRY in the requirements sense.

It takes a fair amount of work to design a good class where one can easily apply the open-closed principle. And one should where it's needed. But tying together two completely unrelated parts of your code base with such a class just because the two pieces of code are almost identical is the wrong approach. Then going ahead and designing it for the open-closed principle merely adds complexity.

Sometimes this is beneficial. Sometimes it isn't.

My point is — more often than not — conventional wisdom is that DRY is always preferable, whereas the reality is not that simple.

In my honest opinion, frequent rewrites are by-and-large a disastrously bad idea, for several reasons. If there is one thing I would change about Google, it would be to slow down the frenetic pace of change inside. Rewrites just make the pace of change untenable. And I say this as one who is totally part of the problem: I helped rewrite significant parts of V8, the JS VM in Chrome, particularly the optimizing JIT compiler, TurboFan. (Don't get me wrong--I am not knocking any one specific project, my coworkers, even my leadership, etc). I've been at Google 9 years, and I don't know how barely any of it works anymore.

1. The assumption that requirements and environment around software change so frequently that it must be burned down to the ground and rewritten is a big part of the problem. Why do the requirements of software change? A. Scale. B. Because the software around it changed. Bingo.

2. Rewrites actively destroy institutional expertise. Instead of learning more as time goes on, engineers' knowledge becomes obsolete as old systems are constantly changing and rewritten for unclear benefit. Experts can no longer rely on their knowledge for more than a couple of years. This is extremely bad for critical pieces of infrastructure. In short, no one ever masters anything. This is due not just to incentives but due to change itself.

3. No one ever has time to do an in-depth followup study on whether the rewritten artifact was better than the original. Instead people go on their gut feeling of having rewritten something they often did not write themselves (and did not fully understand) with something new and shiny of their own creation. The justification of the outcome is done, in short, by the people who have a big vested interest in declaring success. (And yes, me too).

4. The idea that the software requirements keep changing around software is promulgated by the exact same people who never spend any time up front simply writing requirements down. Well, no fracking wonder the requirements seem to change somewhere in the middle or years later: they were never anticipated in the first place! We'd do better overall if the industry in general did some good ole requirements engineering. Most people I talk to have never even heard of this. Instead, we never have any time to stop and think about doing things right, but we always find time to rewrite from scratch.

5. Zero incentive to do things right. As a field, as industry, we are actually not very serious about writing good software. Instead, we're just going to trash it after 5 years. So software is constantly bad. But the next rewrite!

The drive for rewrites is mostly a swindle in my opinion. The reality is that some software needs to be shot in the head, some needs to be rewritten, but most software needs to be just maintained. That means bugfixes, performance improvements, scalability improvements, and sometimes, yes, refactoring too. But bugfixes and incremental performance improvements don't get anyone promoted. Even more cynically, but very realistically, "old" software that is maintained by experts means a dependency on those experts, and they end up being expensive. Corporations hate when their employees have job security! Rewrites are A.) driven by an influx of young talent who want to make their mark, B.) incentivized by the promotion process and C.) driven by a corporate pressure (everywhere, not just Google) to make sure that programmers and software are commoditized to avoid dependencies, bus factor, and job security.

So much 3, 4, and 5.

I was so spoiled at my last employer. Leaving was a mistake, but I can't go back because I relocated. Seems like nobody in this entire town "gets it". Seriously.

What I have noticed from the travel industry is that backend developers tend to write a ton of original code and rarely refactor anything as though they are scared an improvement is always a regression. Frontend developers tended to not refactor anything either, but then they were scared to write any code at all (note: I am a frontend developer).

Part of this fear was justified because their automation was shitty. Often things were properly code reviewed, but changes would only be accepted with the smallest possible footprint. It hurts when the diff is a static text comparison counting every character, which means a bunch of comments explaining things or white space changes looks like a ton of code changes. It also keeps you from removing unnecessary code or reorganizing things. The killer though were frameworks for everything including test automation in various different flavors for the same sorts of things, which is like writing tests for testing of tests. In this case testing became a block to check off that had little or no real value.

But both, search and adwords, have been rewritten multiple times since their launch.

You can have a look at the papers created by Jeff Dean (1) and Sanjay Ghemawat (2) which mention some of the new concepts/technologies/features used in those products.

1: https://ai.google/research/people/jeff 2: https://ai.google/research/people/SanjayGhemawat

https://developers.google.com/google-ads/api/docs/start

Disclosure: I co-author the Python client library and have written a few of the docs on the site listed above.